- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes TRACE-S2S to improve the diagnostic quality and evidence traceability of industrial maintenance reports. The method first converts industrial time series into semantic evidence units (Effects) with variables, time intervals, and unique identifiers, and then trains an evidence tracing policy (ETP) to select evidence for coarse-grained fault diagnosis, fine-grained cause analysis, and maintenance recommendations, respectively. The downstream report reasoner, signal-to-semantic conversion module, and knowledge graph remain frozen, with only the ETP optimized through reinforcement learning; the training reward is composed of fault diagnosis, fine-grained cause, maintenance action quality, and evidence and query costs. The authors evaluate the method on multiphase flow and thermoelectric process datasets and report that it outperforms multiple baselines on diagnostic, maintenance recommendation, and evidence tracing metrics. The paper's core claim is that dynamically organizing evidence for the inference stage can improve the quality of the final report and enable the report's conclusions to be linked to the corresponding observations.
Critical strengths
- The problem is practically meaningful. Industrial maintenance reports need not only to identify faults, but also to explain the specific causes, what maintenance actions should be taken, and the basis for the conclusions. The paper focuses on the complete pipeline from diagnosis to recommendations, rather than merely optimizing classification accuracy.
- The method design fits the staged nature of diagnosis. TRACE-S2S organizes evidence separately for coarse diagnosis and fine-grained cause analysis, and allows later stages to reselect evidence based on prior diagnoses. This is more consistent with real diagnostic workflows than using the same evidence set for all stages or a single relevance ranking.
- Evidence tracing participates in conclusion generation. The selected Effects are provided to the frozen reasoner and citations are retained in the report, so the evidence path can influence report formation, rather than merely supplementing post-hoc explanations. The method also supports querying variables not yet examined under a budget.
- The experimental results show noteworthy potential. In the authors' reported experiments, TRACE-S2S outperforms the compared methods across multiple datasets and evaluation metrics; the ablation studies also show that removing the learned ETP or the downstream reinforcement learning leads to declines in some fine-grained diagnosis, maintenance action, and evidence tracing metrics.
Critical weaknesses
- Limited data scale and generalization evidence. The thermoelectric data contain eight real fault cases. Even if multiple time windows are extracted from these cases, the windows do not necessarily constitute independent fault samples. The current results are therefore insufficient to adequately demonstrate that the method generalizes to different equipment, plants, or operating conditions.
- Evidence citation does not yet sufficiently demonstrate reasoning faithfulness. Citations in the report can be traced back to observations, but this does not necessarily mean that the reasoner actually relied on those observations, or that those observations are sufficient to support the corresponding conclusions. The model may also rely mainly on candidate information or the knowledge graph to answer.
- The ablations of key design choices are not comprehensive enough. The current ablations mainly examine whether GRPO is used and whether a learned ETP is used, and have not yet sufficiently disentangled the individual contributions of staged evidence selection, organizing evidence by candidates, active querying, and hierarchical reward gating.
- Reward evaluation and cost settings still require more thorough validation. The fine-grained cause and maintenance action rewards depend on fixed evaluators and reference annotations; their reliability and agreement with human judgment are not sufficiently clear in the main text. In addition, the normalization of the evidence penalty may be affected by changes in the evidence pool size after querying, and training and inference costs are not fully quantified in the main text.
Questions for authors
- Regarding generalization evaluation: When the thermoelectric data are split into training, validation, and test sets, are the splits strictly isolated by independent fault event or equipment unit? Can you provide sample counts by fault category, equipment, or event, as well as means, variances, and confidence intervals over multiple random splits or multiple runs?
- Regarding evidence faithfulness: Have the authors conducted evidence intervention experiments, such as removing key Effects, or replacing them with irrelevant or contradictory evidence, and observing whether the diagnosis and maintenance recommendations change accordingly? Such experiments would help distinguish between "traceable citations exist in the report" and "the conclusions actually depend on the cited evidence."
- Regarding component contributions: Could you add independent ablations, under the same number of inference calls, evidence counts, and query budgets, for staged evidence selection, candidate-conditioned selection, active querying, and hierarchical reward gating? This would help determine which designs the performance gains come from, rather than from other factors in the overall system.
- Regarding reward and resource costs: How is the fixed evaluator validated — is its semantic matching accuracy or agreement with human judgment sampled and reviewed by experts? Also, how does the normalization denominator of the evidence penalty change after active querying, and has a sensitivity analysis of alternative penalty forms been conducted? Can you report the training cost, per-instance inference latency, and average number of queries?
3 comments
(W2/Q2) Evidence faithfulness and intervention tests
Thank you for asking whether the conclusions depend on the cited observations. We present observation interventions in a representative thermal-power case with the coarse diagnosis held fixed, then examine the resulting shifts in fine-cause predictions and maintenance recommendations to show how the supplied evidence influences both diagnostic reasoning and action selection.
Each intervention was repeated three times. A representative example is thermal-power case07, where we hold the coarse diagnosis fixed at abnormally high winding temperature in the circulating-water-pump motor and modify the fine-stage Effects.
| Observations supplied to the reasoner | Fine-cause prediction |
|---|---|
| Original evidence, including an isolated winding-temperature anomaly | Temperature-measurement wiring fault |
| Remove the isolated sensor anomalies; retain high temperatures across multiple coils and low cooling-water pressure | Insufficient cooling-water pressure |
| Provide high temperatures across multiple coils, low cooling-water flow, and a valve command/feedback mismatch | Low-flow or valve-related abnormality |
| Provide high temperatures across multiple coils and reduced air-cooler heat transfer, with normal water pressure and flow | Air-cooler fouling |
With the coarse diagnosis fixed, the fine-cause explanation changes according to the supplied observations. The maintenance recommendation also shifts from checking the measurement circuit to investigating the cooling system. Changes in pressure, flow, valve feedback, and heat-transfer observations further distinguish different cooling-related mechanisms.
Appendix D.4 also reports coarse-stage deletion, reversal, normalization, and replacement interventions for cases 02, 07, and 08, together with additional fine-stage checks. These tests show how predictions respond to altered observations. We interpret them as evidence-sensitivity checks, rather than a dataset-wide estimate of causal faithfulness.
(W3/Q3) Contributions of individual ETP components
Thank you for asking us to disentangle the contributions of the ETP components. To identify the contribution of each design, we compare separate ablations of staged evidence selection, candidate–Effect associations, active querying, and hierarchical reward gating, and explain their effects on coarse diagnosis, fine-cause accuracy, and maintenance-action quality.
The following results are single-run thermal-power ablations; performance across three training seeds is reported separately in Appendix D.1.
| Variant | Coarse Accuracy (%) | Fine Accuracy (%) | Action F1 (%) |
|---|---|---|---|
| Full TRACE-S2S | 100.00 | 87.50 | 73.33 |
| Reuse coarse-stage evidence for fine reasoning | 100.00 | 53.12 | 64.58 |
| Remove candidate–Effect associations | 78.12 | 65.62 | 59.58 |
| Disable querying | 34.38 | 15.62 | 15.62 |
| Random querying | 62.50 | 43.75 | 41.88 |
| Remove hierarchical reward gating | 93.75 | 46.88 | 48.44 |
The results distinguish the contributions of the four designs:
-
Staged selection: Reusing coarse-stage evidence preserves 100% Coarse Accuracy but reduces Fine Accuracy from 87.50% to 53.12%, showing the value of revisiting the observations for finer-grained reasoning.
-
Candidate associations: Removing candidate–Effect associations reduces both diagnostic accuracy and maintenance-action quality.
-
Active querying: Disabling queries causes the largest decline. Random querying recovers part of the performance but remains below learned querying, supporting the value of selecting which additional variables to inspect.
-
Reward gating: Removing hierarchical gating reduces Fine Accuracy to 46.88% and Action F1 to 48.44%, supporting the hierarchical training objective.
These experiments complement the existing GRPO and learned-policy ablations by examining the individual evidence-selection, organization, acquisition, and reward designs. Appendix D.7 defines the ablation variants and reports their single-run results.
(W4/Q4) Evaluator validation, evidence penalties, and costs
Thank you for requesting clearer evaluator validation, penalty definitions, and computational costs. We clarify reward computation, assess report-scoring reliability through expert review and cross-family rescoring, explain the evidence and query penalties, and report training time and inference latency.
Evaluator validation. We clarify the roles of reference matching and LLM evaluation. The fine-cause reward uses matching against expert-approved acceptable cause IDs. The action reward is computed from action-ID matches, with an additional evaluator check that the recommendation preserves the action’s applicability conditions.
Domain experts manually review 20% of the evaluated reports using the same factuality criteria. We further assess report-scoring sensitivity by rescoring 44 fixed thermal-power reports, comprising 22 reports from each of two seeds, with GPT-6.1-terra and Qwen3.8-max. The reports, reference facts, and scoring procedure remain fixed. Across the two seeds, the absolute difference in seed-level Report F1 between GPT-6.1-terra and DeepSeek-V4-Pro rescoring averages 1.75 percentage points, while Qwen3.8-max assigns higher scores for both seeds. Appendix D.5 provides the report-fact evaluation prompt, model and decoding settings, and cross-family rescoring results. These human and cross-family checks assess offline Report Fact F1. The action-applicability component of the training reward uses a separate evaluator prompt and is outside the scope of these checks.
Evidence-penalty normalization. In Eq. (5), the denominator is the number of Effects in the final visible pool, including those revealed through querying. The normalized term therefore measures the proportion of available evidence selected for reasoning. We separately charge for additional signal inspections: the evidence penalty uses λE = 0.1, and every query incurs a cost of λQ = 0.05, regardless of whether the newly revealed Effects are subsequently selected. This accounts separately for the amount of evidence retained and the cost of inspecting additional recorded signals.
Training and inference costs. ETP training uses a single H100 80GB. The recorded costs are:
| Setting | Training wall-clock time (h) | Inference latency (s) | Average queries |
|---|---|---|---|
| Thermal power, Protocol-S | 2.920 | 12.586 | 0.80 |
| Thermal power, Protocol-U | 7.115, all folds combined | 12.822 | 0.82 |
| Multiphase flow, Protocol-S | 4.802 | 35.785 | 0.74 |
| TEP, Protocol-S | 6.062 | 33.830 | 1.20 |
The thermal-power Protocol-U training time averages 0.889 h per fold. Training times are logged wall-clock measurements, and inference latency estimates are based on the reported two-example timing measurements. At inference, TRACE-S2S generates one deterministic trajectory, without reward-evaluator calls or parameter updates. Appendix D.8 provides the timing, token-usage, and model-call breakdown.
Because the denominator uses the final visible pool, querying can reduce the evidence fraction even when the selected set is unchanged. The separate query cost limits this effect, but does not make the penalty invariant to pool growth. The reported experiments evaluate this specified formulation; a comparison of alternative normalizations is not included.
Response to Reviewer 3 (6yEX)
We thank you for your detailed feedback that helped us strengthen the experimental analysis and clarify the method.
(W1/Q1) Independent fault events and generalization
Thank you for emphasizing independent fault events and evidence of generalization. We address your concerns about independent samples and generalization by explaining event-isolated and chronological splits, clarifying the thermal fault records, reporting three-seed variability, and adding TEP results from independent runs in a different industrial process.
First, Protocol-U strictly separates independent fault events. TRACE-S2S achieves 68.75% Fine Accuracy and 65.83% Action F1 under Protocol-U (Table 3, using the aggregation defined in Appendix C.2). We conduct eight-fold leave-one-case-out evaluation: in each fold, the held-out event’s signals, derived sequences, and report targets are entirely excluded from policy training and validation, which use the remaining seven events.
Protocol-S also respects temporal ordering. For each thermal-power fault sequence, the first 60%, the next 20%, and the final 20% are assigned to training, validation, and testing, respectively. Reporting windows are extracted separately within these partitions, without shared observation timestamps across splits.
Second, we clarify the organization of the thermal-power data. The data comprise separate industrial fault events recorded at different times in the collaborating plant. Each event is represented by a continuous monitoring record spanning days to weeks, preserving the temporal context surrounding the fault. Protocol-U uses the complete event as the unit of separation, excluding its signals, derived sequences, and report targets from training and validation.
Third, we report Protocol-S results across three training seeds. The thermal-power results are shown below as mean ± standard deviation; all values are percentages.
The seed-wise results and detailed sample statistics are provided in Appendix D.1.
Finally, we extend the evaluation to the Tennessee Eastman Process (TEP). This benchmark adds a distinct chemical process, with 80 independent simulation runs across eight diagnostic categories. Each category contains ten runs, divided into six training, two validation, and two test runs, giving 48/16/16 runs overall. Each run belongs entirely to one split.
The thermal-power-trained ETP is applied to TEP without policy finetuning. The target task supplies its process-specific Effects and candidate knowledge.
The policy reaches 58.33% Fine Accuracy and 44.17% Action F1 on this evaluation, while Evidence F1 is 23.11%. These results support partial reuse of the policy across processes, while identifying agreement with target-process evidence annotations as a remaining transfer limitation.