acceptodds
Under review as a conference paper at ICLR 2027

EviDiag: Posterior-Guided Evidence Seeking for Auditable Time-Series Diagnosis

Abstract

Time-series diagnosis commonly maps observed signals directly to a fault or root-cause label, leaving evidence acquisition and decision making implicit. Although large language models (LLMs) have enabled readable diagnostic reasoning, existing approaches typically lack explicit mechanisms for sequentially planning evidence acquisition and quantitatively revising fault beliefs. This challenge is compounded in multivariate time series, where fault evidence is often buried in context-dependent temporal dynamics rather than directly available in text or logs. We present EviDiag, a posterior-guided evidence-seeking agent that turns diagnosis into an auditable process of examination and belief revision. EviDiag recovers a context-conditioned healthy trajectory from a pre-fault anchor to ground subsequent examinations, and organizes diagnosis into a within-case inquiry loop and an across-case learning loop. The within-case inquiry loop uses the fault posterior and finite-horizon information value to guide an LLM planner in selecting the next examination, whose outcome revises the fault belief via Bayesian updating. The across-case loop distills resolved state-action-outcome transitions into retrievable experience and recalibrates evidence likelihoods to improve future inquiry. Across microservice and industrial-process benchmarks, Across microservice and industrial-process benchmarks, EviDiag attains 1.06× and 1.90× the top-1 diagnostic accuracy of the best baseline, respectively, while producing auditable evidence-to-posterior traces.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Time-series diagnosis, Large language models, Evidence seeking, Bayesian belief updating, Auditable diagnosis, Experience-guided inquiry

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_FocwOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper considers a fault diagnosis problem and transforms it into a sequential evidence-seeking problem. It first forecasts the baseline time series and then captures the hit-or-miss pattern on the residual, then uses the idea of Bayes fault posterior update to update the faith over time and then gives the final decision. It involves an LLM to pick the next test from the information gain values. Experiments show its improvement on diagnosis top-1 and produce auditable traces.

Critical strengths
  1. The setup is pretty interesting. Instead of giving the final diagnosis, the paper explores a way to show the sequential updating on the faith for the final diagnosis. This is an important direction for grounded and auditable diagnosis.

  2. The paper is well grounded in methodlogical guidance. The use of information gain test selection and the naive bayes procedure is well motivated under the setup.

  3. The paper does a good calibration and paired significance test for the reported conclusions, which is scientifically sound.

Critical weaknesses
  1. In general, the paper needs more evidence for the complicated components being proposed. One of the central claims about using LLM planning for Bayesian belief update is not well isolated. For the update part, no results show what will happen if we directly let LLm update the belief instead of the formula; for the selection, the tests ran in a fixed order perform on par with the LLM-driven one, meaning LLM choices add no meaningful gain there.

  2. The paper claims the loop can be done online and improve future diagnosis, but no setup supports this promise. A separate setup is needed to actually solidly support this claim otherwise, it shall be retracted.

  3. One of the key parts, the so-called auditable traces produced by the method, is never analyzed. Some evaluations on the traces besides the final diagnsis shall be analyzed to faithfully support that such a produced auditable trace is indeed better than naive Bayes baselines.

  4. Although tests are provdied, but all results come from a single run - the run-to-run variance is unknown. As least certain pilot study is needed to show that the conclusion is not changed with multiple runs.

  5. Minor: Figure 1 does not make clear which module is LLM-based and which is more trained and which is fixed. Making it hard to interpret and understand.

Questions for authors

Please see the weaknesses part.

8 comments

Dear Reviewer Focw,

We sincerely thank you for your encouraging and constructive review. Your suggestions have helped us strengthen the empirical validation of the framework's main components. In particular, we have added controlled comparisons of belief updating and examination selection, introduced a sequential evaluation of diagnostic evolution, and expanded the quantitative analysis of diagnostic traces. We have also supplemented the evaluation with multi-seed results and clarified the module roles in Figure 1. We address your concerns individually below.

W1: Isolating the Contributions of LLM Planning and Explicit Bayesian Belief Updates.

Thank you for this valuable suggestion. We have strengthened the component evaluation with two controlled studies that separately examine explicit Bayesian belief updating and adaptive examination selection.

LLM-based versus Bayesian Belief Updating. To directly assess the contribution of explicit numerical belief updating, we replace the analytical Bayesian update with direct LLM-based belief updating while keeping the examination sequences and observed outcomes fixed. Specifically, we use the same LLM (DeepSeek-V4.1-Flash) as in our main experiments and provide it with the current posterior, selected examination, and corresponding Hit/Miss outcome. The LLM is instructed to update the fault probabilities using its own reasoning capabilities, without relying on an explicit likelihood model. In contrast, EviDiag applies the analytical Bayesian update. The experimental results are presented in Table R11. Beyond diagnostic ranking, we use the multiclass Brier score to assess the accuracy of the estimated posterior probabilities over candidate faults (lower is better).

Table R11. Comparison of analytical Bayesian and LLM-based belief updating under identical examination trajectories.

BenchmarkBelief updateR@1 (%)R@3 (%)Macro-F1 (%)Brier score ↓
RCAEvalAnalytical Bayesian58.3385.0061.040.5800
LLM-based46.6778.3344.800.7316
E-TEPAnalytical Bayesian86.0090.0086.280.2556
LLM-based50.0064.0045.770.7183

As shown in Table R11, the analytical Bayesian update achieves substantially higher diagnostic accuracy and lower Brier scores on both benchmarks. Direct LLM-based updating reduces R@1 from 58.33% to 46.67% on RCAEval and from 86.00% to 50.00% on E-TEP. These results support the use of an explicit probabilistic evidence model for belief updating rather than relying solely on language-based reasoning. This motivates our separation of LLM-guided examination selection from analytical Bayesian belief revision. We have added this controlled comparison to Appendix E.6.4 (p. 36, lines 1922–1939, blue; Table 15, p. 37, lines 1945–1956, red), with a summary in Section 4.4.2 (p. 9, lines 474–479, red).

Adaptive versus fixed-order examination selection. To more systematically evaluate the benefits of adaptive examination selection and reduce the influence of a particular examination order, we have extended the comparison to both randomly ordered and numerically informed fixed-order policies on RCAEval and E-TEP.

Specifically, we construct two types of fixed-order baselines that share the same directional initialization, examination tools, calibrated likelihoods, Bayesian belief updates, and maximum inquiry budget of 10 rounds with EviDiag. The only difference is the subsequent examination-selection strategy: instead of adaptively selecting examinations based on the evolving diagnostic state, both baselines execute examinations according to predefined sequences. Random Fixed-order uses three different examination orders generated with three random seeds, with each order remaining fixed across all test cases within a run. QKQ_{K}-ranked Fixed-order instead ranks the examinations by their QKQ_{K} values immediately after directional initialization and fixes this case-specific order throughout subsequent inquiry. This controlled comparison allows us to assess the contribution of adaptive examination selection while keeping the other diagnostic components unchanged.

Table R12. Round-wise diagnostic accuracy (R@1, %) of adaptive and fixed-order examination selection over the first five inquiry rounds (maximum budget: 10 rounds).

BenchmarkSelection policyInquiry rounds
12345
RCAEvalRandom Fixed-order (Seed 1)40.0036.6738.3335.0043.33
Random Fixed-order (Seed 2)40.0041.6745.0043.3350.00
Random Fixed-order (Seed 3)40.0040.0045.0045.0046.67
QKQ_{K}-ranked Fixed-order40.0051.6756.6758.3353.33
EviDiag40.0056.6758.3358.3358.33
E-TEPRandom Fixed-order (Seed 1)60.0060.0070.0068.0070.00
Random Fixed-order (Seed 2)56.0062.0064.0066.0064.00
Random Fixed-order (Seed 3)60.0060.0060.0060.0064.00
QKQ_{K}-ranked Fixed-order66.0066.0068.0068.0068.00
EviDiag64.0064.0070.0072.0080.00

As shown in Table R12, fixed-order policies can achieve comparable or even higher accuracy during the early inquiry rounds, particularly on E-TEP. However, EviDiag achieves the highest accuracy by the fifth round on both benchmarks, outperforming the best fixed-order policy by 5.00 and 10.00 percentage points on RCAEval and E-TEP, respectively. These results suggest that adaptive examination selection provides additional benefits beyond an informed but predefined examination order. The first-five-round results are reported in Table 20 of Appendix E.6.7 (p. 40, lines 2107–2123, red), under a maximum budget of 10 rounds; the protocol and discussion are provided at p. 40, lines 2144–2159, and p. 41, lines 2176–2177 (blue).

These comparisons provide a more comprehensive assessment of adaptive examination selection across different predefined orders and diagnostic settings. Together with the belief-update ablation, they allow us to distinguish the respective contributions of examination planning and numerical evidence aggregation.

W2: Sequential Evaluation of Cross-Case Diagnostic Evolution.

Thank you for raising this concern. In the original experiments, the historical experience library and calibrated likelihoods were constructed from validation cases and frozen throughout testing. To directly evaluate cross-case diagnostic evolution, we have added a sequential experiment. As our historical-experience ablation shows that both benchmarks benefit from historical information, with more pronounced improvements on E-TEP, we focus on E-TEP to examine how these benefits emerge as confirmed fault cases accumulate.

Sequential diagnostic evaluation. We start with an empty experience library and uncalibrated examination likelihoods. Test cases are randomly ordered and diagnosed sequentially. For each case, the ground-truth label is revealed only after the diagnosis is recorded. The confirmed case is then used to distill and store diagnostic experience, while examination likelihoods are recalibrated after every 10 newly confirmed cases. Thus, each diagnosis uses only historical information from previously confirmed cases.

We compare two settings: No History, which performs neither experience accumulation nor likelihood calibration, and Evolving History, which enables both mechanisms. To assess the robustness of diagnostic evolution to different case orders, we repeat the experiment five times, randomly shuffling the test cases in each run. At each ten-case checkpoint, we report cumulative performance over all cases diagnosed up to that point, averaged across the five runs.

Table R13. Cumulative diagnostic performance on E-TEP under sequential evaluation (mean ± std over five runs).

SettingMetric1020304050
No HistoryR@1 (%)40.00 ± 18.7140.00 ± 3.5437.33 ± 3.6538.50 ± 4.8736.40 ± 1.67
Inquiry rounds9.54 ± 0.509.42 ± 0.229.45 ± 0.059.47 ± 0.139.52 ± 0.08
Evolving HistoryR@1 (%)34.00 ± 5.4837.00 ± 13.0444.67 ± 7.6752.00 ± 7.3753.60 ± 4.77
Inquiry rounds9.50 ± 0.498.73 ± 0.508.10 ± 0.387.26 ± 0.337.04 ± 0.17

Note: Each checkpoint reports cumulative performance over the first n diagnosed cases. Inquiry rounds denote the average number of inquiry rounds per case.

Results and discussion. As shown in Table R13, Evolving History initially performs comparably to No History, but its advantage becomes increasingly apparent as confirmed cases accumulate. From the 30-case checkpoint onward, Evolving History achieves progressively higher cumulative R@1, reaching 53.60 ± 4.77% at the final checkpoint, compared with 36.40 ± 1.67% for No History. Meanwhile, the average number of inquiry rounds progressively decreases to 7.04 ± 0.17, compared with 9.52 ± 0.08 for No History. These results demonstrate that accumulated diagnostic experience and periodic likelihood calibration can improve subsequent diagnoses while reducing the number of examinations required.

This experiment complements our original history-size analysis by evaluating the actual test-then-update process. We have incorporated the sequential evaluation into Appendix E.6.4 (p. 37, lines 1969–1983 and 1992–1997, and p. 38, lines 2031–2033, blue; additional evaluation details at p. 37, lines 1985–1990, light-blue background highlighting; Table 16, p. 37, lines 1958–1966, red; Figure 7, p. 38, lines 1999–2015, light-blue background highlighting), clarifying that it simulates cross-case diagnostic evolution under different case orders rather than chronological online deployment. A summary is provided in Section 4.4.3 (p. 10, lines 491–494, dark red).

W3: Quantitative Evaluation of Auditable Diagnostic Traces.

Thank you for this important suggestion. We have extended the evaluation of EviDiag's diagnostic traces beyond illustrative examples through three complementary analyses: (1) quantitative assessment of examination-level diagnostic contributions, (2) independent expert evaluation of recorded selection rationales, and (3) comparative error attribution against a numerical Bayesian baseline. Together, these analyses examine both the diagnostic progression and the additional information made inspectable by EviDiag's traces, as summarized in Section 4.5 (p. 10, lines 501–520, blue).

(1) EviDiag acquires more beneficial evidence throughout the diagnostic process. To assess the quality of diagnostic traces beyond final accuracy, we compare EviDiag with Active NB-IG and the numerical max-QKQ_{K} policy on RCAEval benchmark. All three methods follow the standardized 10-round protocol, with the same directional initialization, calibrated likelihoods, and stopping criteria.

We evaluate Diagnostic Impact (Eq. 7), which measures changes in the posterior margin of the ground-truth fault over its strongest competitor, together with the proportion of positive-impact examinations and two diagnostic state transitions: Correction (incorrect → correct) and Regression (correct → incorrect). Mean Impact is averaged over the same maximum inquiry budget for all cases entering inquiry, with zero impact assigned to unexecuted rounds. The remaining metrics are computed over executed examinations, with Correction and Regression normalized by the numbers of initially incorrect and correct pre-examination states, respectively.

Reviewer_CWgcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

My read of the paper is that the proposed EviDiag method tries to diagnose faults in multivariate time series by choosing which examination to conduct next. The method first predicts a healthy reference from a pre-alarm anchor, then tests for patterns like steps, ramps, and oscillations in the residual to find match.

An LLM model was used to choose the next examination using the current fault probabilities. The tools then return the examination outcomes, and a Naive Bayes model was then used to update the probabilities. Previously resolved cases were also used to build the experience library and calibrate the examination likelihoods.

The authors evaluated on RCAEval and E-TEP, with this historical information fixed before testing. They reported 58.33% and 76.00% top-1 accuracy, respectively, together with component ablations, inquiry curves, paired statistical tests, calibration results, and diagnostic traces that are helpful.

Critical strengths
  1. I like that the idea that an LLM model can be used to chooses what to examine next, while the tools return the outcomes and a fixed rule updates the fault probabilities. This seems like a reasonable and potentially effective way to make the diagnosis easier to check and more human-in-the-loop. The failed Socket case was particularly useful as tt shows how the thresholds suppress Socket evidence and how later latency examinations shift the diagnosis toward Delay, so I can follow where things go wrong. I find this more informative than just giving a free-form explanation of the final answer.

  2. The E-TEP result is interesting, where the accuracy goes from 40% for the best listed baseline to 76%, and this remains significant in the paired analysis. There are also useful checks in the appendix. The authors splited trajectories and built history only from validation cases, and fixed it before testing. I appreciate the paired tests with correction for multiple comparisons and the posterior calibration results. The authors also acknowledged that the statistical separation on RCAEval is much weaker.

Critical weaknesses
  1. I'm still not clear how the initialization and examination budget are handled in the main comparisons. In the paper's Section 3.1 it starts with a prior over the fault set, but the archived failure trace first runs three direction checks and then keeps and renormalizes only five hypotheses (Figure 8/Table 8). This trace uses a 15-round budget, so it may differ from the five-round main RCAEval experiment. Then in Section E.6.4 it also confirms that hard pruning is used in the evaluated posteriors. Once a fault gets to zero probability, then Equation 4 cannot bring it back. For me this is not clear as to which experiments use these steps, which baselines share them, and do the initial checks count toward the budget?

Also, I saw that some of the controlled results do not match well. At five inquiry rounds, Figure 3's max-Q_K curve is around 55% R@1, while Table 2's "w/o LLM Inquiry (Q_K)" reports 43.33% under the stated five-round RCAEval budget. Maybe these are different controls, initializations, or runs? but I could not find a clear explanation. Figure 4 then gives 71.8% full-history MRR, compared with 70.00% in Table 1. If the figure also uses MRR@3 on the same cases, its 58.33% R@1 and 81.67% R@3 give an upper bound of 70.00%. A different MRR definition could explain this, but it needs to be stated clearly to be understood easily. I would like these detail clarified before making a conclusion about the planner's improvement.

  1. What I am less convinced about is the specific benefit of the retrieving experience. I think the historical-learning experiments show more clearly that the overall system benefits from history. Removing the "historical experience" drops E-TEP accuracy from 76% to 40%, but what exactly is removed here? The history-size experiment rebuilds both the experience library and the calibrated likelihood table, and neither study seems to report calibration-only and retrieval-only results seperately. I would like to understand how much the retrieved lessons benefit in addition to learning the fault-examination probabilities from resolved cases.

Additionally, the probabilistic baselines share the calibrated likelihoods, and replacing the planner with max-Q_K gives 38% on E-TEP. I think that this is useful evidence for the tested planner. My remaining question is how much action selection helps by avoiding related evidence, like step/ramp outcomes from the same signal or P50/P90 latency. The failure audit itself raises this dependence issue. I think a classifier on the same historical binary outcomes, or a numerical policy that accounts for dependence, would then help clarify it. On RCAEval, the lead over CoT is only two of 60 cases and is therefore not statistically significant, so I do not find much additional support there for a broader advantage claim made.

Questions for authors
  1. Could the authors help clarify which main experiments use the direction checks and hard top-five pruning shown in the archived trace? I would like to know the actual algorithm and variable-selection rule, and which baselines share these steps? Do the initial checks count toward the inquiry budget, and how often does pruning remove the true fault?

  2. I'm curious why the five-round max-Q_K result in Figure 3 differs from the "w/o LLM Inquiry" result in Table 2. Were different saved runs or initializations used? Also, what MRR definition gives the 71.8% full-history result in Figure 4, compared with 70.00% MRR@3 in Table 1?

  3. In the "w/o Historical Experience" ablation, does the likelihood table stay fixed when retrieval is removed, or does that change too? Are separate calibration-only and retrieval-only results available? These would hlep me understand what the experience library adds. If there is an existing control using the same binary evidence while accounting for dependence, that would also help interpret the planner gain.

8 comments

Dear Reviewer CWgc,

We sincerely thank you for your careful examination of EviDiag's experimental protocols and results. Your comments have helped us improve the rigor of the experimental comparisons and gain a clearer understanding of the contributions of different components. In particular, we have standardized the initialization, pruning, and inquiry-budget settings, corrected inconsistencies in the reported results, and supplemented the evaluation with targeted experiments on historical experience and action selection. We address your concerns individually below.

Responses to Critical Weaknesses

W1: Clarification of Experimental Protocols and Consistency of Reported Results.

11 Clarification of Initialization, Candidate Pruning, and Inquiry Budgets

Thank you for highlighting these ambiguities. We have audited the original experimental protocols and standardized the initialization, pruning, and examination-budget settings in the revised evaluation.

Original experimental protocol. EviDiag starts from a uniform prior over all candidate fault hypotheses. During initialization, it selects the three variables with the highest health-relative contributions and executes the available directional examinations in both directions. The resulting Hit/Miss outcomes are used to update the initial posterior. EviDiag then retains the five highest-probability hypotheses and renormalizes their probabilities. Subsequent inquiries further remove hypotheses whose posterior probabilities fall below 0.01.

As you correctly noted, these pruning operations are irreversible under Eq. (4). Moreover, the original budgets of 5 rounds on RCAEval and 15 rounds on E-TEP count only the inquiries after initialization, excluding the initial examinations. These initialization and pruning procedures were used in the original EviDiag main experiments and applicable component evaluations, but were not shared by the external baselines. To address these concerns, we have revised the main comparisons and controlled experiments as follows.

Standardized experimental protocol.

(i) Directional initialization. EviDiag starts from a uniform prior over all candidate faults. We calculate each variable's contribution from its squared deviation from the predicted healthy trajectory after the alarm. The contributions are then normalized across variables, and the three highest-scoring variables are selected for directional initialization. For these variables, we execute all available Higher/Lower examinations defined in the knowledge base. The resulting Hit/Miss outcomes update the uniform prior using Eq. (4), with the same calibrated likelihood table as in subsequent inquiries. Previously executed examinations are excluded from later selection. Directional initialization is clarified in Section 3.1 (p. 3, line 161, and p. 4, lines 183–184, blue).

(ii) Candidate retention. We remove both the hard top-five pruning and subsequent low-probability pruning. All candidate fault hypotheses remain available throughout diagnosis. Consequently, the ground-truth fault is never excluded by hard pruning in the revised experiments.

(iii) Inquiry budget and stopping rule. We use a maximum of 10 inquiry rounds on both RCAEval and E-TEP. After at least two inquiry rounds, EviDiag stops when the top-ranked fault remains unchanged across the two most recent posterior updates and its current probability reaches 0.8. Otherwise, inquiry continues until the budget is exhausted or no examination remains. The total examination cost includes both initialization checks and subsequent executed examinations.

The confidence-based stopping rule balances diagnostic accuracy and examination cost rather than guaranteeing a correct diagnosis. On RCAEval, 26 of the 60 test cases (43.33%) terminate before the maximum budget under this rule, with 17 of these diagnoses being correct (65.38%). The average number of inquiry rounds is 7.85 across all test cases.

(iv) Baseline-specific protocols. The controlled Bayesian comparisons share EviDiag's directional initialization, executable examinations, calibrated likelihoods, and stopping rule. External LLM-based baselines retain their original diagnostic mechanisms. Inquiry-style methods (ReAct and OpenRCA) receive the available initialization information and operate under a maximum of 10 additional examinations, while retaining their own tools and stopping criteria. CoT, RAG, and FaultExplainer follow their respective non-interactive diagnostic protocols.

Table R6 summarizes the revised settings and distinguishes the common examination-budget constraints from method-specific diagnostic mechanisms.

Table R6. Comparison of the original and revised experimental protocols.

SettingEviDiag / Active NB-IGPassive NBReAct / OpenRCACoT / RAG / FaultExplainer
Directional initializationShared checks and Bayesian updatesNot requiredInitialization summary provided in promptNot applicable
Examination toolsSharedAll eligible EviDiag toolsReAct: Shared tools. OpenRCA: LLM-generated Python analysesMethod-specific
Calibrated likelihoodsSharedShared——
Hard posterior pruningNoNo——
Additional inquiry budget10Exhaustive10—
Stopping criterionPosterior-basedAll examinations completedMethod-specificMethod-specific
Examination costInitialization + executed inquiriesAll eligible examinationsInitialization evidence + executed examinationsNon-interactive

Note: “—” indicates that the setting is not applicable.

The revised protocol is applied to the updated main comparisons and controlled component evaluations, with method-specific differences explicitly stated in Section 4.1 (p. 7, lines 373–377, blue) and Appendix E.2 (p. 23, lines 1204–1221, blue).

(2) Correction of Inconsistent Experimental Results.

We have re-examined the controlled experiments and identified two reporting inconsistencies. We have standardized the corresponding experimental configurations and metric definitions and revised the affected results and conclusions accordingly.

Figure 3 versus Table 2. In the original manuscript, the discrepancy between the five-round max-QKQ_{K} result in Figure 3 and the original w/o LLM Inquiry result in Table 2 arose from inconsistent likelihood-calibration settings. Figure 3 used historically calibrated fault–examination likelihoods, whereas Table 2 reported results from an uncalibrated run.

To isolate the contribution of the LLM-guided planner, the numerical max-QKQ_{K} control should retain the same calibrated likelihoods as full EviDiag. We have therefore corrected this setting and repeated the comparison under the standardized experimental protocol. Table R7 specifies the configurations of all revised ablations, and Table 2 reports their updated diagnostic results.

The revised comparison retains likelihood calibration when replacing the LLM planner with numerical max-QKQ_{K}, thereby separating the contribution of LLM-guided selection from that of historical likelihood calibration. Figure 3 (p. 9, lines 447–459, discussion in Section 4.4.2, lines 469–472, blue) and Table 2 (p. 8, lines 379–388) now use the same evidence model, and the performance at matched inquiry rounds is reported consistently. The calibrated numerical-planner configuration is specified in Table 14 (also shown as Table R7 below), lines 1891–1902. The corrected comparison narrows the previously reported performance gap and allows us to characterize the contribution of LLM-guided inquiry more accurately, as further discussed in W2.

Table R7. Experimental configurations of the revised component ablations.

VariantAnchored recoveryExperience retrievalCalibrated likelihoodExamination-selection policyNumerical inquiry values (IG/Qk\mathbf{Q}_{\mathbf{k}})
Full EviDiag✓✓✓LLM✓
w/o Anchored Recovery×✓✓LLM✓
w/o Historical Experience✓××LLM✓
w/o LLM Inquiry✓—✓Numerical calculation✓
w/o Information-Gain Values✓✓✓LLM×
Retrieval-only✓✓×LLM✓
Calibration-only✓×✓LLM✓

Note: “—” indicates that action-level experience is not retrieved for the numerical selection policy. All configurations follow the same evaluation protocol except for the components being examined.

Figure 4 versus Table 1. We also identified an inconsistency in the MRR calculation. The original 71.8% full-history result in the original Figure 4 corresponds to untruncated MRR, whereas Table 1 reports MRR@3. Specifically, among the 60 RCAEval cases, the ground-truth fault is ranked first in 35 cases, second in 14 cases, and third in none. The corresponding MRR@3 is therefore (35 + 14/2)/60 = 70.00%. The untruncated MRR additionally assigns reciprocal-rank credit to ground-truth faults ranked below third place.

To make the history-size evaluation directly comparable with the main results, we have standardized all ranking metrics to use the same truncation definitions and evaluation procedure, as specified in Appendix E.3 (p. 23, lines 1225–1234, lines 1234–1238, blue additions). Under the revised protocol, EviDiag achieves an MRR@3 of 70.00% on RCAEval and 87.67% on E-TEP, as reported in the updated Table 1. The history-size analysis has also been revised accordingly in Figure 4 (p. 9, lines 447–459) and Section 4.4.3 (pp. 9–10, lines 483–489).

We apologize for these reporting errors. Beyond correcting the individual discrepancies, we have checked the consistency of the experimental configurations, metric implementations, and result tables throughout the revised manuscript.

W2: Disentangling Historical Experience and Understanding the Benefits of Action Selection.

(1) Disentangling experience retrieval and likelihood calibration.

Thanks for this valuable suggestion, which has helped us clarify the complementary roles of the two mechanisms underlying EviDiag's experience-grounded evolution. We have extended the historical-experience ablation into a controlled two-factor experiment to separately evaluate experience retrieval and likelihood calibration.

In the original w/o Historical Experience variant, both components were disabled. Thus, the reported performance difference reflected their combined contribution rather than distinguishing their individual effects.

To determine how much each mechanism contributes independently, we introduce two additional configurations: Retrieval-only and Calibration-only. In Retrieval-only, action-level experience is retrieved to guide inquiry planning, while the examination likelihoods retain their original uncalibrated values. Conversely, Calibration-only uses historical cases to calibrate the fault-conditioned examination likelihoods P(o∣f,a)P(o \mid f,a), without retrieving action experience for the planner. Full EviDiag uses both mechanisms.

Reviewer_tjT3OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors tackle the problem of fault diagnosis in multivariate time-series, through sequential examination of deviations from a predicted healthy trajectory. They propose EviDiag, which uses an LLM to select temporal patterns to examine, and updates fault probabilities through a Naive Bayes model. The authors evaluate the method on microservice and industrial process benchmarks, finding improved top-1 accuracy.

Critical strengths
  • The authors tackle an important problem in time series anomaly detection and attribution.

  • The method design is intuitive. The LLM only chooses which examination to run, which mitigates hallucination risks.

Critical weaknesses
  1. From the evaluations compared to the baselines, the biggest gain in performance for EviDiag appears to be access to the labeled historical cases, which only EviDiag has access to. In fact, EviDiag does worse than many baselines when the historical cases are removed. The main result therefore appears to depend on retrieved labeled examples that no other baseline can use. For a fair comparison, the authors should evaluate the LLM baselines with the same validation cases supplied as retrieved exemplars.

  2. The authors use a set of fixed temporal examination operators. There are seven such operators, and they seem rather limited in describing how a time-series could change (e.g. frequency changes). In addition, the examinations do not test changes in relationships between variables. This limits the practical effectiveness of the method.

  3. The examinations identify statistical patterns without establishing whether an affected variable is causal to the fault or just correlational. Further, the evaluation assumes a single fault drawn from six or ten predefined categories, with no evaluation of unseen faults or simultaneous faults. These also limit the practical utility of the method. The authors should clarify these limitations.

  4. The Naive Bayes conditional independence assumption seems questionable for some of the examinations tested here. For example, P50 and P90 summarize the same distribution and can thus remain dependent after conditioning on the fault. This is also true for several of the temporal operators like level, step, and ramp, which describe overlapping properties of the signal. Treating these as independent seems like it should result in errors in the posterior estimation.

  5. The paired significance tables in the appendix are hard to parse, but it does not look like the method outperforms most baselines with significance, especially on RCAEval. The authors should add the results of these significance tests (or at least the confidence intervals) to the main results table.

  6. The authors only evaluate on two datasets, and both use synthetically injected faults. There are several ways in this differs from real-world faults, for example, there is only one fault per case, it comes from a fixed catalog that matches the candidate list, and the onset is sharp. Thus, it is unclear how well the method would work in practice.

Questions for authors
  1. Could the authors explain why the passive NB baseline (which uses all possible examinations) does worse than EviDiag (which uses an LLM to select)? Shouldn't the Bayesian model be strictly better with more evidence?
7 comments

Dear Reviewer tjT3,

We sincerely thank you for your careful review and constructive feedback. We appreciate your recognition of the importance of the problem and the intuitive design of EviDiag, particularly the separation of LLM-based examination planning from tool-grounded evidence generation. Your comments raise important questions regarding comparison fairness, probabilistic assumptions, and practical applicability. We address each concern below.

W1: Fair Comparison with Access to Historical Diagnostic Cases.

Thank you for raising this important fairness concern. We have strengthened the baseline comparisons by introducing a matched-history evaluation protocol, in which all methods have access to the same labeled historical cases used by EviDiag. This additional evaluation allows us to distinguish the benefits of historical information from those of EviDiag's experience-grounded diagnostic mechanism.

Historical information for baselines. We agree that EviDiag benefits from historical diagnostic supervision. We therefore provide the same historical case pool to all five LLM-based baselines through retrieved labeled examples. The LLM baselines use these examples according to their respective diagnostic workflows, while EviDiag retrieves action-level experience distilled through reflection. For Passive NB and Active NB-IG, we use the same historical cases and Beta–Bernoulli likelihood calibration as EviDiag. Thus, the methods share the underlying historical information, while retaining their respective mechanisms for utilizing it. We have also standardized the evaluation protocol and clarified the method-specific examination settings, as further detailed in W1 of Reviewer CWgc.

Additional results. Table R1 presents the end-to-end diagnostic performance of all methods under the matched-history evaluation protocol.

Table R1 (Table 1). End-to-end diagnostic ranking results (%) under matched historical-case access.

RCAEval

MethodR@1R@3R@1 CIMRR@3F1
Passive NB56.6783.33[45.00, 68.33]68.6159.69
Active NB-IG56.6786.67[45.00, 68.33]69.4459.69
RAG-based36.6760.00[30.00, 43.33]47.2231.35
CoT41.6778.33[35.00, 48.33]56.3934.65
ReAct48.3388.33[38.33, 58.33]64.7243.97
FaultExplainer20.0033.33[13.33, 26.67]25.5621.65
OpenRCA51.6788.33[41.67, 61.67]68.6148.51
EviDiag58.3385.00[46.67, 70.00]70.0061.04

E-TEP

MethodR@1R@3R@1 CIMRR@3F1
Passive NB80.0092.00[70.00, 90.00]85.6780.33
Active NB-IG80.0090.00[70.00, 90.00]84.6779.48
RAG-based54.0060.00[50.00, 60.00]57.0056.67
CoT62.0072.00[60.00, 66.00]65.3356.78
ReAct52.0070.00[40.00, 64.00]60.3349.89
FaultExplainer42.0062.00[34.00, 50.00]51.3335.38
OpenRCA54.0076.00[44.00, 64.00]64.0052.13
EviDiag86.0090.00[76.00, 94.00]87.6786.28

Note: R@k denotes Recall@k, MRR denotes MRR@3, F1 denotes Macro-F1. CI represents the 95% bootstrap confidence interval for R@1. Bold indicates the best point estimate in each metric column.

As shown in Table R1, EviDiag achieves the highest R@1, MRR@3, and Macro-F1 on both RCAEval and E-TEP, with R@1 of 58.33% and 86.00%, respectively. Compared with the strongest LLM-based baselines, EviDiag improves R@1 by 6.67 and 24.00 percentage points on the two benchmarks. The historically calibrated Bayesian baselines also achieve competitive performance, reaching 56.67% and 80.00% R@1, respectively.

These results provide a more comprehensive comparison under matched access to historical cases. Historical likelihood calibration establishes a strong probabilistic baseline, while EviDiag achieves further numerical gains through its combined diagnostic framework. The contributions of likelihood calibration, retrieved reflection, and LLM-guided inquiry are further examined through controlled ablations and decision-level analyses in our response to W2 of Reviewer CWgc.

The corresponding results and revised protocol have been provided in Section 4.1, lines 369–372 (p. 7, red), Table 1, lines 325–337 (p. 7), Section 4.2, lines 393–403 (p. 8, blue), Appendix E.4, lines 1246–1252 (p. 24, blue).

W2: Scope and Extensibility of the Examination Operators.

Thank you for raising this concern. We have clarified in Section 3.3 that the seven examination operators were selected specifically for the target systems, to characterize the temporal symptoms considered in the two benchmarks. The examination set can be tailored to different systems according to their diagnostic requirements, while retaining EviDiag’s core framework.

Importantly, the seven operators serve as executable diagnostic tools within EviDiag rather than defining a fixed examination vocabulary for all systems. Each tool analyzes the relevant time series and returns a binary Hit/Miss outcome. For other systems, the examination set can be extended according to their specific fault symptoms, for example by incorporating frequency-domain examinations or multivariate relationship checks. Provided that the execution procedures, fault-symptom associations, and fault-conditioned outcome likelihoods of the new examinations are specified, these examinations can be incorporated into EviDiag while retaining the underlying examination-selection and Bayesian-update mechanisms.

We have clarified this tool-based design and its extension in the revised manuscript (Section 3.3, lines 248–251).

W3: Statistical Evidence, Causal Attribution, and Fault-Set Assumptions.

Thank you for highlighting these important distinctions. We have refined the diagnostic formulation to distinguish statistical symptom evidence from causal attribution and to clarify the scope of the diagnosis problem addressed by EviDiag.

Statistical evidence versus causality. In this work, the examinations characterize statistical changes in observed variables and provide symptom evidence for fault diagnosis. Such changes may result from a fault, but the affected variable is not necessarily the root cause of the fault. For example, a CPU-Higher outcome in RCAEval benchmark may occur under CPU stress, memory stress, or other fault types. Similarly, oscillations in reactor cooling-water temperature in E-TEP benchmark may be associated with a cooling-water temperature disturbance or valve stiction fault. EviDiag accumulates evidence from successive examinations to infer the underlying fault type, rather than directly treating an abnormal variable as the root cause.

Diagnostic task and applicability. EviDiag is primarily designed to enable auditable fault diagnosis through sequential evidence acquisition and explicit posterior updates. Its current formulation considers a predefined set of candidate faults under a single-fault setting, consistent with the evaluated benchmark tasks. We have made these assumptions explicit in Section 3.1. Future work could further explore zero-shot learning for unseen faults and joint Bayesian inference over fault combinations for simultaneous faults, to broaden its applicability in real-world industrial settings.

We have added the above-mentioned clarifications in Section 3.1, lines 160–185 (p. 3-4) and Conclusion (lines 532-534, p. 10).

W4: Conditional Independence Assumption and Correlated Diagnostic Evidence.

Thank you for raising this important concern. EviDiag adopts Naive Bayes as its probabilistic evidence model, updating fault posteriors using the likelihoods of binary examination outcomes conditioned on candidate faults. This model introduces a conditional independence approximation to reduce the complexity of probability estimation from limited historical fault cases. As you pointed out, examinations of related signals or overlapping temporal patterns may remain dependent even after conditioning on the fault type. To better understand the practical implications of this approximation, we have conducted two additional analyses: (1) an empirical assessment of conditional dependence between examinations and (2) a sensitivity analysis of its potential impact on posterior estimation and diagnostic performance.

(1) Empirical analysis of examination dependence. To determine whether examination outcomes exhibit conditional dependence in practice, we analyze all 36 historical validation cases in RCAEval. Specifically, we compute the conditional mutual information between binary examination outcomes given the fault label, considering all 120 pairs of the 16 executable examinations.

The results reveal particularly strong dependence among different temporal examinations of the same latency signal. For example, P90-Rise (Ramp) and P90-Rise (Step) exhibit a conditional mutual information of 0.648 bits, while P90-Higher and P90-Rise (Step) exhibit 0.505 bits. We also observe nonzero conditional mutual information between P50 and P90 examinations of the same type (0.145–0.214 bits for Higher, Rise Ramp, and Rise Step), consistent with the potential dependence highlighted in your comment.

Given the limited number of historical fault cases, which is also a practical concern in industrial diagnosis, these estimates may be affected by finite-sample fluctuations. To distinguish statistical dependence from such fluctuations, we further perform a within-fault permutation test. Table R2 presents the six examination pairs with the strongest statistical evidence of dependence, all with unadjusted p-values below 0.01.

Table R2. Conditional dependence between selected examination pairs on RCAEval.

Examination pairConditional mutual information (bits)Raw p-value
P90-RiseRamp / P90-RiseStep0.6480.000037
P90-Higher / P90-RiseStep0.5050.000444
P90-Higher / P90-RiseRamp0.4040.002222
P50-Higher / P50-RiseStep0.3220.003086
P50-RiseRamp / P50-RiseStep0.3250.004630
P50-Higher / P50-RiseRamp0.2690.009259

The strongest dependence, between P90-RiseRamp and P90-RiseStep, remains statistically significant after Holm correction across all 120 comparisons (adjusted p ≈ 0.0044). We treat the six selected pairs as potentially dependent examinations for the following sensitivity analysis, rather than claiming statistically established dependence for all six pairs.

(2) Posterior sensitivity to correlated evidence. Having identified potentially dependent examinations, we further investigate how strongly their contributions affect the estimated fault posteriors and diagnostic results. Given the limited historical sample size, reliably estimating fault-conditioned joint likelihoods for examination combinations remains challenging. We therefore conduct a controlled evidence-weighting experiment to explore the effect of potentially overlapping evidence, rather than introducing a fully dependence-aware probabilistic model.

Specifically, we replay the diagnostic trajectories of 60 RCAEval test cases, keeping the examination sequences, Hit/Miss outcomes, likelihoods, and stopping positions fixed. To account for potentially overlapping evidence, we introduce a weight into the original Bayesian update in Eq. (4) as
br+1(f)∝br(f) P(or∣f,ar)wrb_{r + 1}(f) \propto b_{r}(f)\, P\left( o_{r} \mid f,a_{r} \right)^{w_{r}}

When the current examination forms one of the six potentially dependent examination pairs with a previously executed examination, we set wr=λw_{r} = \lambda. Otherwise, wr=1w_{r} = 1. We consider three settings: λ=1\lambda = 1 (original Naive Bayes update), 0.50.5 (partially discounting the likelihood contribution), and 00 (omitting the likelihood contribution of the potentially dependent examination).

The motivation for this weighting is that, when two examinations provide overlapping information, the subsequent examination may contribute less additional diagnostic evidence than assumed by the factorized likelihood model. The three weights therefore allow us to examine posterior sensitivity to progressively stronger reductions in the contribution of potentially redundant evidence.

Beyond diagnostic ranking, we evaluate the multiclass Brier score to assess the accuracy of the estimated posterior probabilities over candidate faults. We additionally examine changes in the top-ranked fault relative to the original update, distinguishing corrections, regressions, and transitions between incorrect predictions.

Table R3. Sensitivity of diagnostic posteriors to the weighting of potentially dependent evidence.

Evidence weightR@1 (%)Brier score ↓Changed Top-1Wrong→TrueTrue→WrongWrong→another Wrong
1.058.330.5800000
0.560.000.5325212
0.060.000.51310541

Note: R@1 is calculated over all 60 test cases, while the multiclass Brier score is evaluated on the 58 cases with valid diagnostic posteriors. All diagnostic transitions are measured relative to the original update (λ = 1.0).

Comments on the paper

1 comment
AuthorsOpenReview ↗

Global Response: Summary of Major Revisions

We sincerely thank the Area Chair and all three Reviewers for their thoughtful evaluations and constructive suggestions. We are particularly pleased that Reviewer CWgc appreciated our LLM–Bayesian design and found the diagnostic traces and experimental findings informative, while Reviewer CWgc considered our sequential evidence-seeking formulation interesting and well-grounded for auditable diagnosis. We also greatly appreciate Reviewer tjT3's recognition of the importance of our research problem and the intuitive design of our framework. The constructive suggestions from all three Reviewers have helped us substantially strengthen our experimental evidence, clarify our methodological contributions, and draw more rigorous conclusions. We have carefully revised the manuscript accordingly.

Below, we summarize five major improvements:

  • More Rigorous and Fair Evaluation. We standardized the experimental protocols, matched historical information across methods, and strengthened statistical reporting. Under these settings, EviDiag achieves the highest R@1, MRR@3, and Macro-F1 point estimates on both benchmarks, while we have carefully refined our statistical claims.

  • Clearer Roles of Historical Experience. New controlled ablations reveal that likelihood calibration provides the primary gains from historical information, while retrieved reflection offers complementary benefits for examination planning. This substantially clarifies how our two experience-evolution mechanisms work together.

  • Stronger Evidence for LLM–Bayesian Collaboration. Additional experiments on fixed-order planning, direct LLM belief updating, and evidence redundancy help establish a clearer division of responsibilities: LLMs support context-sensitive examination selection, while explicit Bayesian inference ensures evidence-grounded belief updating.

  • Deeper Analysis of Diagnostic Traces and Robustness. We introduced examination-level metrics, independent expert assessments, matched failure-case analysis, and multiple-run evaluations.

  • Validation on Real-Flight UAV Data. We extended EviDiag to the ALFA dataset and evaluated 11 fault-containing flight sequences, demonstrating its application to a different physical system through representative diagnostic cases.

Overall, these revisions have helped us move beyond end-to-end performance comparisons toward a more comprehensive understanding of why EviDiag works and how its components complement one another. We have incorporated the corresponding changes into the manuscript and provided detailed responses to each Reviewer.

We sincerely appreciate your time and consideration, and hope these additional efforts help address your concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.