- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper considers a fault diagnosis problem and transforms it into a sequential evidence-seeking problem. It first forecasts the baseline time series and then captures the hit-or-miss pattern on the residual, then uses the idea of Bayes fault posterior update to update the faith over time and then gives the final decision. It involves an LLM to pick the next test from the information gain values. Experiments show its improvement on diagnosis top-1 and produce auditable traces.
Critical strengths
-
The setup is pretty interesting. Instead of giving the final diagnosis, the paper explores a way to show the sequential updating on the faith for the final diagnosis. This is an important direction for grounded and auditable diagnosis.
-
The paper is well grounded in methodlogical guidance. The use of information gain test selection and the naive bayes procedure is well motivated under the setup.
-
The paper does a good calibration and paired significance test for the reported conclusions, which is scientifically sound.
Critical weaknesses
-
In general, the paper needs more evidence for the complicated components being proposed. One of the central claims about using LLM planning for Bayesian belief update is not well isolated. For the update part, no results show what will happen if we directly let LLm update the belief instead of the formula; for the selection, the tests ran in a fixed order perform on par with the LLM-driven one, meaning LLM choices add no meaningful gain there.
-
The paper claims the loop can be done online and improve future diagnosis, but no setup supports this promise. A separate setup is needed to actually solidly support this claim otherwise, it shall be retracted.
-
One of the key parts, the so-called auditable traces produced by the method, is never analyzed. Some evaluations on the traces besides the final diagnsis shall be analyzed to faithfully support that such a produced auditable trace is indeed better than naive Bayes baselines.
-
Although tests are provdied, but all results come from a single run - the run-to-run variance is unknown. As least certain pilot study is needed to show that the conclusion is not changed with multiple runs.
-
Minor: Figure 1 does not make clear which module is LLM-based and which is more trained and which is fixed. Making it hard to interpret and understand.
Questions for authors
Please see the weaknesses part.
8 comments
Table R12. Round-wise diagnostic accuracy (R@1, %) of adaptive and fixed-order examination selection over the first five inquiry rounds (maximum budget: 10 rounds).
| Benchmark | Selection policy | Inquiry rounds | ||||
|---|---|---|---|---|---|---|
| 1 | 2 | 3 | 4 | 5 | ||
| RCAEval | Random Fixed-order (Seed 1) | 40.00 | 36.67 | 38.33 | 35.00 | 43.33 |
| Random Fixed-order (Seed 2) | 40.00 | 41.67 | 45.00 | 43.33 | 50.00 | |
| Random Fixed-order (Seed 3) | 40.00 | 40.00 | 45.00 | 45.00 | 46.67 | |
| -ranked Fixed-order | 40.00 | 51.67 | 56.67 | 58.33 | 53.33 | |
| EviDiag | 40.00 | 56.67 | 58.33 | 58.33 | 58.33 | |
| E-TEP | Random Fixed-order (Seed 1) | 60.00 | 60.00 | 70.00 | 68.00 | 70.00 |
| Random Fixed-order (Seed 2) | 56.00 | 62.00 | 64.00 | 66.00 | 64.00 | |
| Random Fixed-order (Seed 3) | 60.00 | 60.00 | 60.00 | 60.00 | 64.00 | |
| -ranked Fixed-order | 66.00 | 66.00 | 68.00 | 68.00 | 68.00 | |
| EviDiag | 64.00 | 64.00 | 70.00 | 72.00 | 80.00 |
As shown in Table R12, fixed-order policies can achieve comparable or even higher accuracy during the early inquiry rounds, particularly on E-TEP. However, EviDiag achieves the highest accuracy by the fifth round on both benchmarks, outperforming the best fixed-order policy by 5.00 and 10.00 percentage points on RCAEval and E-TEP, respectively. These results suggest that adaptive examination selection provides additional benefits beyond an informed but predefined examination order. The first-five-round results are reported in Table 20 of Appendix E.6.7 (p. 40, lines 2107–2123, red), under a maximum budget of 10 rounds; the protocol and discussion are provided at p. 40, lines 2144–2159, and p. 41, lines 2176–2177 (blue).
These comparisons provide a more comprehensive assessment of adaptive examination selection across different predefined orders and diagnostic settings. Together with the belief-update ablation, they allow us to distinguish the respective contributions of examination planning and numerical evidence aggregation.
W2: Sequential Evaluation of Cross-Case Diagnostic Evolution.
Thank you for raising this concern. In the original experiments, the historical experience library and calibrated likelihoods were constructed from validation cases and frozen throughout testing. To directly evaluate cross-case diagnostic evolution, we have added a sequential experiment. As our historical-experience ablation shows that both benchmarks benefit from historical information, with more pronounced improvements on E-TEP, we focus on E-TEP to examine how these benefits emerge as confirmed fault cases accumulate.
Sequential diagnostic evaluation. We start with an empty experience library and uncalibrated examination likelihoods. Test cases are randomly ordered and diagnosed sequentially. For each case, the ground-truth label is revealed only after the diagnosis is recorded. The confirmed case is then used to distill and store diagnostic experience, while examination likelihoods are recalibrated after every 10 newly confirmed cases. Thus, each diagnosis uses only historical information from previously confirmed cases.
We compare two settings: No History, which performs neither experience accumulation nor likelihood calibration, and Evolving History, which enables both mechanisms. To assess the robustness of diagnostic evolution to different case orders, we repeat the experiment five times, randomly shuffling the test cases in each run. At each ten-case checkpoint, we report cumulative performance over all cases diagnosed up to that point, averaged across the five runs.
Table R13. Cumulative diagnostic performance on E-TEP under sequential evaluation (mean ± std over five runs).
| Setting | Metric | 10 | 20 | 30 | 40 | 50 |
|---|---|---|---|---|---|---|
| No History | R@1 (%) | 40.00 ± 18.71 | 40.00 ± 3.54 | 37.33 ± 3.65 | 38.50 ± 4.87 | 36.40 ± 1.67 |
| Inquiry rounds | 9.54 ± 0.50 | 9.42 ± 0.22 | 9.45 ± 0.05 | 9.47 ± 0.13 | 9.52 ± 0.08 | |
| Evolving History | R@1 (%) | 34.00 ± 5.48 | 37.00 ± 13.04 | 44.67 ± 7.67 | 52.00 ± 7.37 | 53.60 ± 4.77 |
| Inquiry rounds | 9.50 ± 0.49 | 8.73 ± 0.50 | 8.10 ± 0.38 | 7.26 ± 0.33 | 7.04 ± 0.17 |
Note: Each checkpoint reports cumulative performance over the first n diagnosed cases. Inquiry rounds denote the average number of inquiry rounds per case.
Results and discussion. As shown in Table R13, Evolving History initially performs comparably to No History, but its advantage becomes increasingly apparent as confirmed cases accumulate. From the 30-case checkpoint onward, Evolving History achieves progressively higher cumulative R@1, reaching 53.60 ± 4.77% at the final checkpoint, compared with 36.40 ± 1.67% for No History. Meanwhile, the average number of inquiry rounds progressively decreases to 7.04 ± 0.17, compared with 9.52 ± 0.08 for No History. These results demonstrate that accumulated diagnostic experience and periodic likelihood calibration can improve subsequent diagnoses while reducing the number of examinations required.
This experiment complements our original history-size analysis by evaluating the actual test-then-update process. We have incorporated the sequential evaluation into Appendix E.6.4 (p. 37, lines 1969–1983 and 1992–1997, and p. 38, lines 2031–2033, blue; additional evaluation details at p. 37, lines 1985–1990, light-blue background highlighting; Table 16, p. 37, lines 1958–1966, red; Figure 7, p. 38, lines 1999–2015, light-blue background highlighting), clarifying that it simulates cross-case diagnostic evolution under different case orders rather than chronological online deployment. A summary is provided in Section 4.4.3 (p. 10, lines 491–494, dark red).
W3: Quantitative Evaluation of Auditable Diagnostic Traces.
Thank you for this important suggestion. We have extended the evaluation of EviDiag's diagnostic traces beyond illustrative examples through three complementary analyses: (1) quantitative assessment of examination-level diagnostic contributions, (2) independent expert evaluation of recorded selection rationales, and (3) comparative error attribution against a numerical Bayesian baseline. Together, these analyses examine both the diagnostic progression and the additional information made inspectable by EviDiag's traces, as summarized in Section 4.5 (p. 10, lines 501–520, blue).
(1) EviDiag acquires more beneficial evidence throughout the diagnostic process. To assess the quality of diagnostic traces beyond final accuracy, we compare EviDiag with Active NB-IG and the numerical max- policy on RCAEval benchmark. All three methods follow the standardized 10-round protocol, with the same directional initialization, calibrated likelihoods, and stopping criteria.
We evaluate Diagnostic Impact (Eq. 7), which measures changes in the posterior margin of the ground-truth fault over its strongest competitor, together with the proportion of positive-impact examinations and two diagnostic state transitions: Correction (incorrect → correct) and Regression (correct → incorrect). Mean Impact is averaged over the same maximum inquiry budget for all cases entering inquiry, with zero impact assigned to unexecuted rounds. The remaining metrics are computed over executed examinations, with Correction and Regression normalized by the numbers of initially incorrect and correct pre-examination states, respectively.
Dear Reviewer Focw,
We sincerely thank you for your encouraging and constructive review. Your suggestions have helped us strengthen the empirical validation of the framework's main components. In particular, we have added controlled comparisons of belief updating and examination selection, introduced a sequential evaluation of diagnostic evolution, and expanded the quantitative analysis of diagnostic traces. We have also supplemented the evaluation with multi-seed results and clarified the module roles in Figure 1. We address your concerns individually below.
W1: Isolating the Contributions of LLM Planning and Explicit Bayesian Belief Updates.
Thank you for this valuable suggestion. We have strengthened the component evaluation with two controlled studies that separately examine explicit Bayesian belief updating and adaptive examination selection.
LLM-based versus Bayesian Belief Updating. To directly assess the contribution of explicit numerical belief updating, we replace the analytical Bayesian update with direct LLM-based belief updating while keeping the examination sequences and observed outcomes fixed. Specifically, we use the same LLM (DeepSeek-V4.1-Flash) as in our main experiments and provide it with the current posterior, selected examination, and corresponding Hit/Miss outcome. The LLM is instructed to update the fault probabilities using its own reasoning capabilities, without relying on an explicit likelihood model. In contrast, EviDiag applies the analytical Bayesian update. The experimental results are presented in Table R11. Beyond diagnostic ranking, we use the multiclass Brier score to assess the accuracy of the estimated posterior probabilities over candidate faults (lower is better).
Table R11. Comparison of analytical Bayesian and LLM-based belief updating under identical examination trajectories.
As shown in Table R11, the analytical Bayesian update achieves substantially higher diagnostic accuracy and lower Brier scores on both benchmarks. Direct LLM-based updating reduces R@1 from 58.33% to 46.67% on RCAEval and from 86.00% to 50.00% on E-TEP. These results support the use of an explicit probabilistic evidence model for belief updating rather than relying solely on language-based reasoning. This motivates our separation of LLM-guided examination selection from analytical Bayesian belief revision. We have added this controlled comparison to Appendix E.6.4 (p. 36, lines 1922–1939, blue; Table 15, p. 37, lines 1945–1956, red), with a summary in Section 4.4.2 (p. 9, lines 474–479, red).
Adaptive versus fixed-order examination selection. To more systematically evaluate the benefits of adaptive examination selection and reduce the influence of a particular examination order, we have extended the comparison to both randomly ordered and numerically informed fixed-order policies on RCAEval and E-TEP.
Specifically, we construct two types of fixed-order baselines that share the same directional initialization, examination tools, calibrated likelihoods, Bayesian belief updates, and maximum inquiry budget of 10 rounds with EviDiag. The only difference is the subsequent examination-selection strategy: instead of adaptively selecting examinations based on the evolving diagnostic state, both baselines execute examinations according to predefined sequences. Random Fixed-order uses three different examination orders generated with three random seeds, with each order remaining fixed across all test cases within a run. QK-ranked Fixed-order instead ranks the examinations by their QK values immediately after directional initialization and fixes this case-specific order throughout subsequent inquiry. This controlled comparison allows us to assess the contribution of adaptive examination selection while keeping the other diagnostic components unchanged.