acceptodds
Under review as a conference paper at ICLR 2027

Trace Before You Conclude: Learning Evidential Reasoning for Industrial Maintenance Report Generation

Abstract

The signals-to-semantics (S2S) paradigm bridges the modality gap between industrial time series and large language models (LLMs) by translating sensor dynamics into textual descriptions for maintenance report generation. This semantic interface, however, leaves a distinct learning problem: how to trace report claims to supporting observations and how to leverage that evidence to improve root-cause analysis and maintenance recommendations. We introduce TRACE-S2S, a framework that grounds a domain-structured reasoning chain in temporal semantic evidence. The reasoning chain consists of coarse root-cause identification, fine-grained cause analysis, and maintenance action recommendation. Guided by an external maintenance knowledge graph, an Evidence Tracing Policy constructs stage-specific evidence packages that directly condition a frozen reasoner. The resulting reports preserve explicit links from their conclusions to supporting observations and knowledge. Evidence tracing therefore controls how conclusions are formed, rather than merely explaining them afterward. We optimize only the evidence policy through downstream reinforcement learning, leveraging fault labels and sparse expert report annotations to reward accurate root-cause diagnosis and appropriate maintenance actions. Training requires no annotations of evidence links, and both the S2S front-end and report reasoner remain frozen. The policy can additionally inspect previously uninspected signals when evidence is insufficient. We evaluate TRACE-S2S on a multiphase-flow process and a real-world thermal-power process, assessing diagnostic accuracy, maintenance-action quality, report factuality, and evidence traceability. On average, TRACE-S2S achieves the evidence tracing F1 of 71.69%, outperforming the strongest baseline by 31.50%.

Primary area:
interpretability and explainable AI
Keywords:
Evidence Tracing, Time Series Reasoning, Temporal Semantics, Large Language Models, Industrial Maintenance

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_6yEXOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TRACE-S2S to improve the diagnostic quality and evidence traceability of industrial maintenance reports. The method first converts industrial time series into semantic evidence units (Effects) with variables, time intervals, and unique identifiers, and then trains an evidence tracing policy (ETP) to select evidence for coarse-grained fault diagnosis, fine-grained cause analysis, and maintenance recommendations, respectively. The downstream report reasoner, signal-to-semantic conversion module, and knowledge graph remain frozen, with only the ETP optimized through reinforcement learning; the training reward is composed of fault diagnosis, fine-grained cause, maintenance action quality, and evidence and query costs. The authors evaluate the method on multiphase flow and thermoelectric process datasets and report that it outperforms multiple baselines on diagnostic, maintenance recommendation, and evidence tracing metrics. The paper's core claim is that dynamically organizing evidence for the inference stage can improve the quality of the final report and enable the report's conclusions to be linked to the corresponding observations.

Critical strengths
  • The problem is practically meaningful. Industrial maintenance reports need not only to identify faults, but also to explain the specific causes, what maintenance actions should be taken, and the basis for the conclusions. The paper focuses on the complete pipeline from diagnosis to recommendations, rather than merely optimizing classification accuracy.
  • The method design fits the staged nature of diagnosis. TRACE-S2S organizes evidence separately for coarse diagnosis and fine-grained cause analysis, and allows later stages to reselect evidence based on prior diagnoses. This is more consistent with real diagnostic workflows than using the same evidence set for all stages or a single relevance ranking.
  • Evidence tracing participates in conclusion generation. The selected Effects are provided to the frozen reasoner and citations are retained in the report, so the evidence path can influence report formation, rather than merely supplementing post-hoc explanations. The method also supports querying variables not yet examined under a budget.
  • The experimental results show noteworthy potential. In the authors' reported experiments, TRACE-S2S outperforms the compared methods across multiple datasets and evaluation metrics; the ablation studies also show that removing the learned ETP or the downstream reinforcement learning leads to declines in some fine-grained diagnosis, maintenance action, and evidence tracing metrics.
Critical weaknesses
  • Limited data scale and generalization evidence. The thermoelectric data contain eight real fault cases. Even if multiple time windows are extracted from these cases, the windows do not necessarily constitute independent fault samples. The current results are therefore insufficient to adequately demonstrate that the method generalizes to different equipment, plants, or operating conditions.
  • Evidence citation does not yet sufficiently demonstrate reasoning faithfulness. Citations in the report can be traced back to observations, but this does not necessarily mean that the reasoner actually relied on those observations, or that those observations are sufficient to support the corresponding conclusions. The model may also rely mainly on candidate information or the knowledge graph to answer.
  • The ablations of key design choices are not comprehensive enough. The current ablations mainly examine whether GRPO is used and whether a learned ETP is used, and have not yet sufficiently disentangled the individual contributions of staged evidence selection, organizing evidence by candidates, active querying, and hierarchical reward gating.
  • Reward evaluation and cost settings still require more thorough validation. The fine-grained cause and maintenance action rewards depend on fixed evaluators and reference annotations; their reliability and agreement with human judgment are not sufficiently clear in the main text. In addition, the normalization of the evidence penalty may be affected by changes in the evidence pool size after querying, and training and inference costs are not fully quantified in the main text.
Questions for authors
  • Regarding generalization evaluation: When the thermoelectric data are split into training, validation, and test sets, are the splits strictly isolated by independent fault event or equipment unit? Can you provide sample counts by fault category, equipment, or event, as well as means, variances, and confidence intervals over multiple random splits or multiple runs?
  • Regarding evidence faithfulness: Have the authors conducted evidence intervention experiments, such as removing key Effects, or replacing them with irrelevant or contradictory evidence, and observing whether the diagnosis and maintenance recommendations change accordingly? Such experiments would help distinguish between "traceable citations exist in the report" and "the conclusions actually depend on the cited evidence."
  • Regarding component contributions: Could you add independent ablations, under the same number of inference calls, evidence counts, and query budgets, for staged evidence selection, candidate-conditioned selection, active querying, and hierarchical reward gating? This would help determine which designs the performance gains come from, rather than from other factors in the overall system.
  • Regarding reward and resource costs: How is the fixed evaluator validated — is its semantic matching accuracy or agreement with human judgment sampled and reviewed by experts? Also, how does the normalization denominator of the evidence penalty change after active querying, and has a sensitivity analysis of alternative penalty forms been conducted? Can you report the training cost, per-instance inference latency, and average number of queries?
3 comments

Response to Reviewer 3 (6yEX)

We thank you for your detailed feedback that helped us strengthen the experimental analysis and clarify the method.

(W1/Q1) Independent fault events and generalization

Thank you for emphasizing independent fault events and evidence of generalization. We address your concerns about independent samples and generalization by explaining event-isolated and chronological splits, clarifying the thermal fault records, reporting three-seed variability, and adding TEP results from independent runs in a different industrial process.

First, Protocol-U strictly separates independent fault events. TRACE-S2S achieves 68.75% Fine Accuracy and 65.83% Action F1 under Protocol-U (Table 3, using the aggregation defined in Appendix C.2). We conduct eight-fold leave-one-case-out evaluation: in each fold, the held-out event’s signals, derived sequences, and report targets are entirely excluded from policy training and validation, which use the remaining seven events.

Protocol-S also respects temporal ordering. For each thermal-power fault sequence, the first 60%, the next 20%, and the final 20% are assigned to training, validation, and testing, respectively. Reporting windows are extracted separately within these partitions, without shared observation timestamps across splits.

Second, we clarify the organization of the thermal-power data. The data comprise separate industrial fault events recorded at different times in the collaborating plant. Each event is represented by a continuous monitoring record spanning days to weeks, preserving the temporal context surrounding the fault. Protocol-U uses the complete event as the unit of separation, excluding its signals, derived sequences, and report targets from training and validation.

Third, we report Protocol-S results across three training seeds. The thermal-power results are shown below as mean ± standard deviation; all values are percentages.

DatasetCoarse AccuracyFine AccuracyAction F1
Thermal power96.88 ± 3.1369.79 ± 15.4265.00 ± 7.64

The seed-wise results and detailed sample statistics are provided in Appendix D.1.

Finally, we extend the evaluation to the Tennessee Eastman Process (TEP). This benchmark adds a distinct chemical process, with 80 independent simulation runs across eight diagnostic categories. Each category contains ten runs, divided into six training, two validation, and two test runs, giving 48/16/16 runs overall. Each run belongs entirely to one split.

TEP evaluationCoarse Accuracy (%)Fine Accuracy (%)Action F1 (%)Evidence F1 (%)
Protocol-S, mean ± SD over three seeds85.42 ± 3.6170.83 ± 7.2256.46 ± 5.6063.18 ± 6.07
Protocol-U, mean 3 folds100.0083.3366.6767.78

The thermal-power-trained ETP is applied to TEP without policy finetuning. The target task supplies its process-specific Effects and candidate knowledge.

The policy reaches 58.33% Fine Accuracy and 44.17% Action F1 on this evaluation, while Evidence F1 is 23.11%. These results support partial reuse of the policy across processes, while identifying agreement with target-process evidence annotations as a remaining transfer limitation.

(W2/Q2) Evidence faithfulness and intervention tests

Thank you for asking whether the conclusions depend on the cited observations. We present observation interventions in a representative thermal-power case with the coarse diagnosis held fixed, then examine the resulting shifts in fine-cause predictions and maintenance recommendations to show how the supplied evidence influences both diagnostic reasoning and action selection.

Each intervention was repeated three times. A representative example is thermal-power case07, where we hold the coarse diagnosis fixed at abnormally high winding temperature in the circulating-water-pump motor and modify the fine-stage Effects.

Observations supplied to the reasonerFine-cause prediction
Original evidence, including an isolated winding-temperature anomalyTemperature-measurement wiring fault
Remove the isolated sensor anomalies; retain high temperatures across multiple coils and low cooling-water pressureInsufficient cooling-water pressure
Provide high temperatures across multiple coils, low cooling-water flow, and a valve command/feedback mismatchLow-flow or valve-related abnormality
Provide high temperatures across multiple coils and reduced air-cooler heat transfer, with normal water pressure and flowAir-cooler fouling

With the coarse diagnosis fixed, the fine-cause explanation changes according to the supplied observations. The maintenance recommendation also shifts from checking the measurement circuit to investigating the cooling system. Changes in pressure, flow, valve feedback, and heat-transfer observations further distinguish different cooling-related mechanisms.

Appendix D.4 also reports coarse-stage deletion, reversal, normalization, and replacement interventions for cases 02, 07, and 08, together with additional fine-stage checks. These tests show how predictions respond to altered observations. We interpret them as evidence-sensitivity checks, rather than a dataset-wide estimate of causal faithfulness.

(W3/Q3) Contributions of individual ETP components

Thank you for asking us to disentangle the contributions of the ETP components. To identify the contribution of each design, we compare separate ablations of staged evidence selection, candidate–Effect associations, active querying, and hierarchical reward gating, and explain their effects on coarse diagnosis, fine-cause accuracy, and maintenance-action quality.

The following results are single-run thermal-power ablations; performance across three training seeds is reported separately in Appendix D.1.

VariantCoarse Accuracy (%)Fine Accuracy (%)Action F1 (%)
Full TRACE-S2S100.0087.5073.33
Reuse coarse-stage evidence for fine reasoning100.0053.1264.58
Remove candidate–Effect associations78.1265.6259.58
Disable querying34.3815.6215.62
Random querying62.5043.7541.88
Remove hierarchical reward gating93.7546.8848.44

The results distinguish the contributions of the four designs:

  • Staged selection: Reusing coarse-stage evidence preserves 100% Coarse Accuracy but reduces Fine Accuracy from 87.50% to 53.12%, showing the value of revisiting the observations for finer-grained reasoning.

  • Candidate associations: Removing candidate–Effect associations reduces both diagnostic accuracy and maintenance-action quality.

  • Active querying: Disabling queries causes the largest decline. Random querying recovers part of the performance but remains below learned querying, supporting the value of selecting which additional variables to inspect.

  • Reward gating: Removing hierarchical gating reduces Fine Accuracy to 46.88% and Action F1 to 48.44%, supporting the hierarchical training objective.

These experiments complement the existing GRPO and learned-policy ablations by examining the individual evidence-selection, organization, acquisition, and reward designs. Appendix D.7 defines the ablation variants and reports their single-run results.

(W4/Q4) Evaluator validation, evidence penalties, and costs

Thank you for requesting clearer evaluator validation, penalty definitions, and computational costs. We clarify reward computation, assess report-scoring reliability through expert review and cross-family rescoring, explain the evidence and query penalties, and report training time and inference latency.

Evaluator validation. We clarify the roles of reference matching and LLM evaluation. The fine-cause reward uses matching against expert-approved acceptable cause IDs. The action reward is computed from action-ID matches, with an additional evaluator check that the recommendation preserves the action’s applicability conditions.

Domain experts manually review 20% of the evaluated reports using the same factuality criteria. We further assess report-scoring sensitivity by rescoring 44 fixed thermal-power reports, comprising 22 reports from each of two seeds, with GPT-6.1-terra and Qwen3.8-max. The reports, reference facts, and scoring procedure remain fixed. Across the two seeds, the absolute difference in seed-level Report F1 between GPT-6.1-terra and DeepSeek-V4-Pro rescoring averages 1.75 percentage points, while Qwen3.8-max assigns higher scores for both seeds. Appendix D.5 provides the report-fact evaluation prompt, model and decoding settings, and cross-family rescoring results. These human and cross-family checks assess offline Report Fact F1. The action-applicability component of the training reward uses a separate evaluator prompt and is outside the scope of these checks.

Evidence-penalty normalization. In Eq. (5), the denominator is the number of Effects in the final visible pool, including those revealed through querying. The normalized term therefore measures the proportion of available evidence selected for reasoning. We separately charge for additional signal inspections: the evidence penalty uses λE = 0.1, and every query incurs a cost of λQ = 0.05, regardless of whether the newly revealed Effects are subsequently selected. This accounts separately for the amount of evidence retained and the cost of inspecting additional recorded signals.

Training and inference costs. ETP training uses a single H100 80GB. The recorded costs are:

SettingTraining wall-clock time (h)Inference latency (s)Average queries
Thermal power, Protocol-S2.92012.5860.80
Thermal power, Protocol-U7.115, all folds combined12.8220.82
Multiphase flow, Protocol-S4.80235.7850.74
TEP, Protocol-S6.06233.8301.20

The thermal-power Protocol-U training time averages 0.889 h per fold. Training times are logged wall-clock measurements, and inference latency estimates are based on the reported two-example timing measurements. At inference, TRACE-S2S generates one deterministic trajectory, without reward-evaluator calls or parameter updates. Appendix D.8 provides the timing, token-usage, and model-call breakdown.

Because the denominator uses the final visible pool, querying can reduce the evidence fraction even when the selected set is unchanged. The separate query cost limits this effect, but does not make the penalty invariant to pool growth. The reported experiments evaluate this specified formulation; a comparison of alternative normalizations is not included.

Reviewer_RFjsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents an industrial maintenance report generation framework designed to adaptively select temporal evidence to support multi-stage diagnostic reasoning. The core component, an Evidence Tracing Policy (ETP), dynamically retrieves relevant evidence to guide coarse- and fine-grained root-cause analysis as well as actionable maintenance recommendations. The ETP is optimized using Group Relative Policy Optimization (GRPO) driven by outcome-based rewards derived from ground-truth fault labels and domain expert annotations. Evaluated across a thermal-power process and a multiphase-flow process, the framework demonstrates noticeable improvements in diagnostic accuracy, maintenance-action quality, evidence traceability, and report factuality compared to baseline methods.

Critical strengths
  1. Grounded Reasoning Mechanism: The paper effectively targets evidence-grounded reasoning by introducing a GRPO-optimized policy to adaptively retrieve relevant temporal signals, addressing a critical challenge in reliable industrial maintenance report generation.

  2. Empirical Performance: Comprehensive experiments on two physical process datasets demonstrate consistent gains over multiple fault diagnosis and generation baselines across several metrics, supporting the utility of the proposed evidence-tracing strategy.

Critical weaknesses
  1. Unclear Contribution of the Maintenance Knowledge Graph: The framework relies on a fixed maintenance knowledge graph that encapsulates structured domain knowledge (e.g., mappings between coarse-grained causes, fine-grained causes, and maintenance actions). However, its exact construction pipeline, scale, and schema are insufficiently detailed. Because this prior knowledge strongly constrains the reasoning space, it is difficult to determine how much performance gain stems from the knowledge graph versus the ETP itself. To isolate the contribution of the ETP, the authors should provide ablation studies removing or perturbing the knowledge graph and clarify whether all baselines were provided identical access to this structured prior knowledge.

  2. Impact of Reasoning Backbone Capability: The core motivation of the ETP is to filter and select pertinent evidence rather than passing all raw temporal evidence to the frozen reasoner. While the authors provide theoretical and empirical support for this design, it remains unclear whether an explicit ETP is strictly necessary when using more capable or larger reasoning backbones. Specifying the exact reasoning backbone architecture and parameter size used in the experiments—along with evaluations across backbones of varying capabilities—would clarify the generalizability and necessity of an explicit policy module.

Questions for authors
  1. Cross-Domain Transferability: Can an ETP trained on one industrial dataset be transferred zero-shot (or with minimal adaptation) to another process without full retraining? How robust is the learned policy across distinct industrial processes and novel fault signatures?

  2. Dataset Scale and Diversity: Given that evaluations are conducted on two industrial processes with a relatively constrained set of unique fault cases, have the authors evaluated the framework on additional benchmarks or real-world industrial datasets to further substantiate its generalization capabilities?

2 comments

Response to Reviewer 2 (RFjs)

We thank you for your thoughtful comments that helped us clarify the framework’s contribution and scope.

(W1) KG construction and contribution

Thank you for highlighting the need to separate knowledge and policy contributions. We describe how the KG is constructed and distinguish its minimal candidate structure from detailed fault knowledge, then use retrained mini-KG ablations to separate the contributions of domain knowledge and learned evidence selection.

KG composition and construction. Our KG contains two components. The first defines the fault-cause and maintenance-action candidates, which we call the mini-KG. This is the minimal task structure needed for industrial reasoning: it specifies the admissible diagnostic and maintenance options; removing it would leave the LLM to generate causes and actions freely. The remaining KG provides additional fault-related knowledge, including measurement-specific symptoms and detailed fault mechanisms. This additional knowledge constitutes most of the graph: for the thermal-power dataset, the mini-KG contains 103 nodes and 114 edges, whereas the full maintenance KG contains 1,030 nodes and 2,590 edges. We construct the KG by extracting triples from plant documentation, authoritative textbooks, and specialist engineering manuals.

New KG ablation. To address your concern, we added the following ablation, retraining ETP under the mini-KG setting. These are single-run thermal-power ablations; performance across three training seeds is reported separately in Appendix D.1.

SettingCoarse Accuracy (%)Fine Accuracy (%)Action F1 (%)
Full KG + ETP100.0087.5073.33
Mini-KG + ETP90.6350.0053.96
Mini-KG without ETP75.006.254.17

Under the same mini-KG, ETP improves Fine Accuracy by 43.75 percentage points and Action F1 by 49.79 points. These gains demonstrate the contribution of learned evidence selection even when only the necessary candidate structure is retained. The full KG further improves performance, showing that detailed domain knowledge and the learned evidence policy make complementary contributions.

(W2) ETP with stronger reasoning backbones

Thank you for asking how stronger reasoning models affect ETP's value. We clarify the separate roles of the 8B evidence policy and frozen reasoner and add comparisons with three alternative direct-generation systems, while distinguishing these comparisons from a controlled test of ETP with alternative reasoning backbones.

Our setup separates an 8B evidence policy from the frozen reasoning model. The policy is Qwen3-8B, optimized with LoRA; the reasoner is DeepSeek-V4-Flash.

We have added direct-generation comparisons with Claude-haiku-4.5, Qwen3.8-max, and GPT-6.1-terra. The thermal-power results are:

Method / backboneFine Accuracy (%)Action F1 (%)
Claude-haiku-4.5, direct generation43.7550.42
Qwen3.8-max, direct generation18.7516.67
GPT-6.1-terra, direct generation6.258.33
TRACE-S2S69.79 ± 15.4265.00 ± 7.64

TRACE-S2S results are reported as mean ± standard deviation across three training seeds. Compared with the best of these three additional direct-generation models, TRACE-S2S improves mean Fine Accuracy by 26.04 percentage points and mean Action F1 by 14.58 points. These results compare complete systems with different reasoning backbones; they do not isolate ETP's incremental benefit under an identical alternative reasoner. The learned-policy ablation in Figure 2 separately assesses ETP with our frozen DeepSeek-V4-Flash reasoner, while Appendix D.7 examines its individual components.

(Q1) Cross-process transfer without retraining

Thank you for asking whether the learned policy transfers across processes. We apply the ETP trained on thermal power directly to TEP without finetuning and report diagnosis, action, and evidence metrics, providing a direct test of whether the learned policy can be reused across industrial processes.

The thermal-power-trained ETP is directly applied to the TEP chemical process without policy finetuning. The target task supplies its process-specific Effects and candidate knowledge. The following results are macro-averaged, and all values are percentages. Fine Accuracy and Action F1 are averaged over the six categories with available reference scores; G2 and G3 are excluded from these two metrics.

TransferCoarse Acc.Fine Acc.Action F1Evidence PEvidence REvidence F1
Thermal power → TEP62.5058.3344.1723.3324.4423.11

The transferred policy achieves 58.33% Fine Accuracy and 44.17% Action F1 on the categories included in these two metrics. These results support partial reuse of the learned evidence policy across industrial processes. The 23.11% Evidence F1 indicates that matching target-process evidence annotations remains a transfer limitation. The transfer setup and detailed results are provided in Appendix D.2.

(Q2) Dataset scale and diversity

Thank you for asking us to broaden the experimental coverage. We broaden the evaluation with TEP, a chemical process with additional fault categories, and report three-seed Protocol-S results plus a three-fold Protocol-U subset to address both dataset diversity and the consistency of the experimental evidence.

The benchmark contains 8 coarse diagnostic categories and 15 fine categories.

We report Protocol-S results across three training seeds and a Protocol-U subset analysis over 3 folds:

TEP evaluationCoarse Accuracy (%)Fine Accuracy (%)Action F1 (%)Evidence F1 (%)
Protocol-S, mean ± SD over three seeds85.42 ± 3.6170.83 ± 7.2256.46 ± 5.6063.18 ± 6.07
Protocol-U mean over 3 folds100.0083.3366.6767.78

The Protocol-U subset scores are arithmetic means across the three specified folds. These experiments extend the evaluation to a chemical process and provide additional evidence from independently generated fault trajectories.

Reviewer_beviOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

TRACE-S2S, a framework that grounds a domain-structured reasoning chain in temporal semantic evidence which provides coarse root-cause identification, fine-grained cause analysis, and maintenance action recommendation. A Qwen3-8B generated policy selects candidate-conditioned temporal “Effects,” inspects additional signals, and constructs separate evidence packages for coarse diagnosis and fine-cause/action reasoning. An external maintenance knowledge graph (KG) supplies candidate causes and procedures. The policy is optimized with GRPO from diagnosis and report outcomes; evidence-link labels are used only for evaluation.

This problem is important in practice and design appears reasonable and provide a claim-to-effect path. The results are based on 2 large industrial data sets with knowledge covered hold out. However, more systematic testing on additional data sets would strengthen the claims.

  • The thermal evaluation has only eight underlying cases, and Protocol-S splits each case’s sequence across train and test. It is unclear how one avoid train /test leakage.
  • the specification of the LLM evaluators is insufficient.
  • The proposed method’s Table 2 evidence precision/recall/F1 triplet is arithmetically inconsistent under the paper’s own definition.

The framework is promising, but the evaluation is too small and under-specified for the strength of the claims, and a headline metric row contains an unresolved inconsistency.

Critical strengths
  • important application class that addresses the need for connecting sensor behavoior to diagnosis and action in maintainence reoprts - this provides the root cause analysis beyond jsut providing a label for a tttern,
  • the architecture of the system is well motivated and design of different components such as frozen signal translator, evidence policy, KG, and reasoner is modular and well conceived
  • the hierarchical approach of establishing a coarse diagnosis followed by a fine diagnosis with evidence is sensible
  • Good data design and use - Effect IDs retain variables and temporal intervals, and the generated report records claim-to-Effect and KG references. ETP does not require evidence-link annotations for training; downstream labels and sparse expert reports supply outcome rewards. Three experts annotate SUPPORT/IRRELEVANT links, with reported Fleiss’ κ=0.951.
  • comprehensive baseline families for comparisons spanning diagnosis-only methods, direct report generators, full-context prompting, ReAct-style retrieval, and a Search-R1-style learned retriever.
  • TRACE-S2S improves fine-cause, action, report-fact, and evidence metrics on the reported tasks, not only coarse accuracy.
Critical weaknesses
  • A fundaemental issue is case level leakage. Protocol-S uses a traditional approach of splitting the time windows into 60/20/20 but this does ensure that there is no temporal semantic leakage of evidence traces. This ista specially a risk given that the data h cases. as only 8 thermal fault cases so it is very likely that train and test are on the same fault

  • The claim of causal and evidential reasoning is not supported- the results measure traceable selected context from within the KG. Since the training and reasoning is constrained by the external knowledge graph the learning essentially could force weak temporal signals to existing causes in the KG. The F1 scores are measuring the overlap with expert links (from the KG) not evidence-link truth.
    To test for evidential reasoning more sufficiency and comprehesiveness analysis is needed - for each claim swapping effects across cases, corrupting vales/descriptions, providing counterevidence to measure claim specific changes

  • The thermal task has 8 case but the details for multiphase task is not provided - number of cases, variables, windows, and test unit. Tables present two-decimal differences without confidence intervals, hypothesis tests, or seed variation - GRPO with a few trajectories with LoRA can be sensitive to seeds. Report per-case results, confusion matrices, bootstrap intervals at the independent-event level instead of window level, and at least a few training seeds.

  • The use of LLM evaluators for semantic matching might create a positive bias. SInce the F1 is based on same family - DeepSeek-V4_pro and generation used DeepSeek-V4_Flash; This setup could favor the wording and style produced from reasoner.
    Report:

  • full prompts and model/API versions;

  • temperature and decoding;

  • number and selection of human-reviewed examples;

  • blinded human-vs-judge agreement;

  • robustness to an independent evaluator family.

  • critical implementation details for the baselines comparisons are missing- prompts, candidate/KG access, number of retrieval/query steps, maximum visible Effects, reward functions, token budgets, training updates, checkpoint rules, and hyperparameter searches.

  • Does Search-R1-style receive the same hierarchical state, candidate associations, KG subtree, query budget, and reward terms?

  • Does ReAct-Query have an equal call/tool budget?

  • Does S2S-KG-Full receive all variables or only initially visible Effects?

  • Is ChatTS-SFT trained with the same test candidate IDs and structured targets? Matched-compute/matched-information comparison is needed for attributing gains specifically to stage-specific evidence tracing.

Questions for authors
  • an important element missing from the current analysis is cause-effect data that is outside the KG - this could help distinguish whether the GRPO training that is based only on learning evidence traces in KG leads to bias in the F1 results. such analysis could strengthen the claiims
  • is a comparison with the use of frontier LLMs available ? much of the GRPO training might be obviated with the latest LLMs and might help establish evidence links that are not enumerated ina KG
4 comments

Response to Reviewer 1 (bevi)

We thank you for your careful review and constructive suggestions on the evaluation protocols and experimental evidence.

(W1) Case-level leakage and data splits

Thank you for raising the concern about case-level leakage. We address your leakage concern through complete fault-event isolation under Protocol-U, chronological splitting before window extraction under Protocol-S, and additional TEP results with independent runs assigned wholly to separate data splits.

First, Protocol-U explicitly separates entire fault events. We use leave-one-case-out evaluation: in each fold, all signals, derived sequences, and report targets from the held-out event are excluded from ETP training and validation. Policy training and selection use only the remaining events. Table 3 reports 68.75% Fine Accuracy and 65.83% Action F1 under this protocol.

Second, Protocol-S uses chronological partitioning before window extraction. For each thermal-power fault sequence, the first 60%, the next 20%, and the final 20% are assigned to training, validation, and testing, respectively. Reporting windows are extracted separately within these partitions, so windows from different splits do not share observed timestamps. This design follows established industrial fault-diagnosis practice: Wang et al. [R1] use chronological training/validation/test partitioning, and Xu et al. [R2] explicitly examine partitioning before overlapping sampling to prevent shared-window leakage. Protocol-S evaluates later observations of known fault scenarios, while Protocol-U evaluates generalization with complete events held out. We have clarified this distinction in the revised protocol description.

Third, we add a benchmark with independently generated fault trajectories. Our Tennessee Eastman Process (TEP) evaluation includes 80 independent simulation runs, organized into eight coarse diagnostic categories and 15 fine categories. Each coarse category contains ten runs, split into six training, two validation, and two test runs, giving 48/16/16 runs overall. Each run belongs entirely to one split. Across three training seeds, TRACE-S2S achieves 70.83 ± 7.22% Fine Accuracy and 56.46 ± 5.60% Action F1. The run-level split and complete results are provided in Appendix D.2.

References

[R1] Y. Wang et al., “Coarse-to-Fine: Progressive Knowledge Transfer-Based Multitask Convolutional Neural Network for Intelligent Large-Scale Fault Diagnosis,” IEEE Transactions on Neural Networks and Learning Systems, 34(2), 761–774, 2023.

https://doi.org/10.1109/TNNLS.2021.3100928

[R2] X. Xu et al., “MS-DRT: A Multilevel and Multiscale Branch Learning Scheme for Fault Diagnosis of Rotating Machinery,” IEEE Transactions on Industrial Informatics, 20(2), 2799–2811, 2024.

https://doi.org/10.1109/TII.2023.3295426

(W2) Evidence grounding and faithfulness

Thank you for asking how conclusions depend on the observed evidence. We clarify the expert evidence annotations and their separation from the training reward, then combine observation interventions with a retrained mini-KG ablation to test evidence dependence and the policy’s contribution when detailed KG knowledge is removed.

The reference evidence links are annotated by domain experts from the recorded signals and canonical Effects. GRPO rewards downstream diagnosis and action outcomes, with evidence/query cost penalties; neither evidence-link labels nor Evidence F1 is used as a training reward.

For thermal-power case07, we fix the coarse diagnosis at abnormally high winding temperature in the circulating-water-pump motor and modify the fine-stage observations:

Observations supplied to the reasonerFine-cause prediction
Original evidence, including an isolated winding-temperature anomalyTemperature-measurement wiring fault
Remove the isolated sensor anomalies; retain high temperatures across multiple coils and low cooling-water pressureInsufficient cooling-water pressure
Provide high temperatures across multiple coils, low cooling-water flow, and a valve command/feedback mismatchLow-flow or valve-related abnormality
Provide high temperatures across multiple coils and reduced air-cooler heat transfer, with normal water pressure and flowAir-cooler fouling

With the coarse diagnosis fixed, the fine-cause conclusion changes with the diagnostic observations. The maintenance recommendation also changes from checking the measurement circuit to investigating the cooling system. These interventions show that the observations influence both the specific fault explanation and the recommended maintenance response. Each intervention was repeated three times; Appendix D.4 provides the intervention inputs and corresponding outputs.

We further test the contribution of learned evidence selection after removing detailed KG knowledge. The mini-KG retains only the cause/action candidate structure, removing measurement-specific symptoms, detailed fault mechanisms, and other additional fault knowledge. In this single-run ablation, ETP retrained under the mini-KG achieves 50.00% Fine Accuracy and 53.96% Action F1, compared with 6.25% and 4.17% without ETP. These gains show that the learned policy remains useful when the detailed fault knowledge is removed. Appendix D.3 provides the complete KG ablation.

Appendix D.4 also reports coarse-stage deletion, reversal, normalization, and replacement interventions for cases 02, 07, and 08, together with additional fine-stage checks. These tests show how predictions respond to altered observations. We interpret them as evidence-sensitivity checks, rather than a dataset-wide estimate of causal faithfulness.

(W3) Dataset details and statistical reporting

Thank you for requesting clearer dataset statistics and seed-wise reporting. We document the multiphase-flow dataset’s size and evaluation unit, then report three-seed means and standard deviations across all three datasets to make both the scale of evaluation and the observed variation in performance explicit.

First, we clarify the multiphase-flow dataset size and evaluation unit. The dataset comprises 13 original experimental runs covering five fault classes, with 24 measured variables and 130 diagnostic windows. Each run contains one annotated fault interval spanning 486–8,970 sampling points. The fault window is the scoring unit used for the reported metrics. The 130 diagnostic windows are derived from 13 original runs and should not be interpreted as 130 independent experiments. The detailed dataset organization is provided in Appendix D.1.

Second, we report results across three training seeds for all three datasets under Protocol-S. The following table presents mean ± standard deviation; all values are percentages.

DatasetCoarse AccuracyFine AccuracyAction F1
Thermal power96.88 ± 3.1369.79 ± 15.4265.00 ± 7.64
Multiphase flow94.87 ± 4.4479.49 ± 4.4471.79 ± 4.44
TEP85.42 ± 3.6170.83 ± 7.2256.46 ± 5.60

These results make the variation across training seeds explicit alongside the average performance. Appendix D.1 provides the seed-wise results and detailed dataset statistics.

(W4) Evaluator reliability and model-family bias

Thank you for raising the concern about evaluator bias. To address evaluator reproducibility, agreement with expert judgments, and possible model-family bias, we provide the scoring configuration, a human verification study, and rescoring of fixed reports by independent evaluator families.

Evaluator specification and reproducibility. The frozen reasoner uses DeepSeek-V4-Flash, while the LLM-based evaluation client uses DeepSeek-V4-Pro. Evaluator calls use temperature = 0, thinking disabled, and JSON-object output. The output limit is 4,096 tokens in the main scoring pipeline and 6,144 tokens in the standalone report-rescoring script. The evaluator returns structured semantic judgments, while the final metrics are computed by Python code. Appendix D.5 provides the report-fact evaluation prompt, output schema, and decoding settings.

Human verification. Domain experts manually reviewed 20% of the evaluated reports using the same factuality criteria. On this subset, the mean absolute difference between human and automated Report F1 was 10.2 percentage points. Appendix D.5 describes the human-review procedure and paired-score calculation.

Independent-evaluator check. We additionally rescore the same 44 thermal-power reports, with 22 reports from each of two seeds, using DeepSeek-V4-Pro as the reference evaluator and GPT-6.1-terra and Qwen3.8-max as evaluators from other model families. The reports, reference facts, and score-aggregation procedure remain fixed across the three evaluators.

Report evaluatorSeed 20260908: Report F1 (%)Seed 20260909: Report F1 (%)
DeepSeek-V4-Pro, rescoring57.5252.01
GPT-6.1-terra60.7151.71
Qwen3.8-max66.0158.90

Relative to DeepSeek-V4-Pro rescoring, GPT-6.1-terra changes Report F1 by +3.19 and −0.30 percentage points for the two seeds, respectively, while Qwen3.8-max assigns scores higher by +8.49 and +6.89 points. These comparisons provide a cross-family check of report scoring and make the sensitivity of absolute Report F1 to evaluator choice explicit.

(W5) Baseline fairness and implementation details

Thank you for requesting clearer controls in the baseline comparisons. We clarify the shared data, information access, and computational controls for the baselines and add targeted component ablations, so the comparisons can distinguish the contribution of learned evidence organization from differences in experimental setup.

All methods use the same data splits and task definitions. The settings for the four baselines are as follows:

  • Search-R1-style receives the same hierarchical state, candidate associations, retrieved KG subtree, query budget, and reward terms as TRACE-S2S. It also uses the same Qwen3-8B policy backbone and frozen reasoner. The comparison therefore focuses on how the policy retrieves and organizes evidence.

  • ReAct-Query accesses the same candidate/KG information and signal-query interface, with matching limits on tool queries and reasoner calls.

  • S2S-KG-Full receives all Effects in the shared visible pool, together with the same retrieved KG context. “Full” means retaining the complete available pool, whereas TRACE-S2S constructs stage-specific evidence subsets from it.

  • ChatTS-SFT uses the same candidate vocabulary and structured output schema, with supervision drawn from the same permitted training partition.

Appendix D.9 specifies the prompt inputs and output interfaces, evidence-access controls and query budgets, policy and baseline training settings, and the separation of training, validation, and test data. The complete report-fact evaluator prompt and its JSON schema are provided in Appendix D.5.

We also add direct ablations of stage-specific evidence selection and candidate–Effect associations. The following table reports single-run thermal-power ablations; the three-seed results are reported in our response to W3.

VariantCoarse Accuracy (%)Fine Accuracy (%)Action F1 (%)
Full TRACE-S2S100.0087.5073.33
Reuse coarse-stage evidence for fine reasoning100.0053.1264.58
Remove candidate–Effect associations78.1265.6259.58

Reusing coarse-stage evidence preserves 100% Coarse Accuracy, but reduces Fine Accuracy by 34.38 percentage points. Removing candidate–Effect associations reduces Fine Accuracy by 21.88 points and Action F1 by 13.75 points. These results support the contribution of selecting evidence separately for fine reasoning and associating it with the relevant candidates. Appendix D.7 provides the complete component-ablation results.

(Q1) Cause–effect relationships beyond the KG

Thank you for suggesting a closer examination of dependence on the KG. We separate the minimal cause/action candidates from detailed KG knowledge and retrain ETP on that reduced graph, testing whether the policy still improves diagnosis and maintenance decisions when predefined symptoms and mechanisms are removed.

The resulting mini-KG contains only the fault-cause and maintenance-action candidates and their minimal task structure, which defines the admissible diagnostic and maintenance options. Measurement-specific symptoms, detailed fault mechanisms, and the remaining fault-related knowledge are removed. For thermal power, the mini-KG contains 103 nodes and 114 edges, compared with 1,030 nodes and 2,590 edges in the full maintenance KG. The full KG is constructed by extracting triples from plant documentation, authoritative textbooks, and specialist engineering manuals.

We retrain ETP under the mini-KG setting. The following table reports single-run thermal-power ablations; the three-seed results are reported in our response to W3.

SettingCoarse Accuracy (%)Fine Accuracy (%)Action F1 (%)
Full KG + ETP100.0087.5073.33
Mini-KG + ETP90.6350.0053.96
Mini-KG without ETP75.006.254.17

Under the same mini-KG, ETP improves Fine Accuracy by 43.75 percentage points and Action F1 by 49.79 points. These gains show that the policy contributes to diagnosis and maintenance decisions even after measurement-specific symptoms and detailed mechanism knowledge are removed. The full KG further improves performance, demonstrating complementary contributions from domain knowledge and learned evidence selection. Appendix D.3 provides the KG construction and ablation details.

(Q2) Comparison with frontier LLMs

Thank you for asking about comparisons with frontier LLMs. We clarify the roles of our evidence policy and frozen reasoner and add comparisons with three alternative direct-generation systems, reporting their system-level performance alongside controlled evidence-policy ablations.

Our framework uses a Qwen3-8B evidence policy optimized with LoRA and a frozen DeepSeek-V4-Flash reasoner. The thermal-power results are:

Method / backboneFine Accuracy (%)Action F1 (%)
Claude-haiku-4.5, direct generation43.7550.42
Qwen3.8-max, direct generation18.7516.67
GPT-6.1-terra, direct generation6.258.33
TRACE-S2S69.79 ± 15.4265.00 ± 7.64

TRACE-S2S results are mean ± standard deviation over three training seeds. Compared with the best of these three additional direct-generation models, TRACE-S2S improves mean Fine Accuracy by 26.04 percentage points and mean Action F1 by 14.58 points. These results compare complete systems with different reasoning backbones; they do not isolate ETP's incremental benefit under an identical alternative reasoner. The learned-policy ablation in Figure 2 separately assesses ETP with our frozen DeepSeek-V4-Flash reasoner, while Appendix D.7 examines its individual components.

The mini-KG experiment in our response to Q1 separately shows that learned evidence selection remains useful after detailed fault knowledge is removed. Appendix D.6 reports the direct-generation results and clarifies the scope of this comparison.

(Summary comment) Evidence precision, recall, and F1

Thank you for highlighting the need to clarify metric aggregation. We explain how separate class-wise averaging accounts for the apparent mismatch between F1 and the reported precision and recall, and add a note to Table 2 to make the aggregation explicit.

Each metric is first computed within each class using the stated definitions and then averaged separately across classes. Consequently, the reported macro-F1 generally differs from the harmonic mean of the reported macro-precision and macro-recall. We have added an explicit note to Table 2 to clarify this aggregation procedure.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.