- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces a dataset for evaluating systems that continuously monitor a motor with a thermal camera, detect faults as soon as they occur, and identify the fault type. Specifically, the ThermalFault dataset records fault occurrence and recovery periods at fixed timestamps while the motor operates under three conditions, namely normal operation, cooling-fan failure, and single-phase undervoltage, at two frequencies of 40 Hz and 50 Hz. The development set contains a total of 21,556 frames, comprising 15,556 normal frames, 3,000 cooling-fan-fault frames, and 3,000 undervoltage frames. The dataset supports a range of tasks, including frame classification, anomaly detection, event detection, recovery monitoring, and post-trigger diagnosis. The experiments cover supervised classification, normal-only anomaly detection, and causal event detection and diagnosis, and are conducted with a variety of baselines.
Critical strengths
The paper establishes a benchmark for thermal motor fault monitoring that brings together session-disjoint splits, causal detection, event recall, detection delay, false-alarm rate, and conditional diagnosis accuracy within a single evaluation protocol, enabling multiple tasks to be assessed under common conditions. Furthermore, by incorporating the recovery process itself into the evaluation, the benchmark extends to a task that is not covered by existing thermal motor datasets.
Critical weaknesses
-
The scale and diversity of the dataset appear too limited for a benchmark. Although the dataset contains a large number of frames, they are sampled at 3-second intervals and are therefore highly correlated rather than independent. The effective number of independent observations is thus closer to the number of sessions (18 development and 3 external), which is small for a dataset contribution. The number of fault events is also very limited, and the recording protocol is highly uniform, with every development fault session following the same schedule and a single severity per fault type. This may bias the learned models and leaves many unobserved fault scenarios uncovered. The complexity of the data is also limited, since all sessions are recorded with the same motor, camera, viewpoint, and procedure. In addition, details on the OCR-based temperature recovery are missing, so it is unclear how accurate the OCR is or how closely the recovered values match actual temperatures, which weakens the reliability of the data. Representative thermal images for each condition are not shown, and no statistics such as means, variances, or distributions are reported to characterize the data. As a result, the paper’s contribution as a dataset paper appears weak.
-
The experimental evaluation has several limitations. With only 12 fault events, the sample size is too small, and the statistical reliability is limited by the absence of confidence intervals, hypothesis tests, and standard deviations for the key results. There is no ablation of the proposed Stage 1 detector, so the contribution of each evidence route remains unclear. The practicality of the method is also not assessed, as there is no evaluation of per-frame computational cost, inference speed, or robustness to changes in real-world conditions such as different motors, ambient temperatures, and camera positions. In addition, systematic analysis of the results, such as interpreting failure cases or identifying why undervoltage is difficult to detect, is limited, and the anomaly detection experiments do not include change-point detection or deep learning-based anomaly detection methods. The writing also needs revision, including an incorrect table caption and unclear metric definitions. Moreover, T2 and T3 use the same metrics and differ only in whether the method is trained on normal data alone, which makes T2 closer to a methodological constraint within the same task than a separate task. The paper also lacks analysis of key parameters and of how sensitive the results are to the chosen thresholds and settings. Finally, the effect of class imbalance is not discussed, and per-class accuracy is not reported.
Questions for authors
- Can the author clearly demonstrate the distinct advantages and effectiveness of the proposed dataset compared to existing datasets?
- Can the author present all the metrics mentioned in the Critical Weaknesses?
- Can the author clearly identify the strengths, weaknesses, differences, and similarities by comparing the actual sample with other datasets?
2 comments
Official comment by fQaP
Q1: The concerns about Q1 (scale, diversity, number of events, fixed protocols, trained models, OCR relability, and statistical analysis) has not been fully addressed.
Q2 : The analysis of statistical reliability lacks detail, and the ablation is not provided. Furthermore, failure cases and challenges of low-voltage detection also insufficient. These concerns still remain unresolved.
Therefore, I keep my original rating.
Response to Reviewer fQaP
We thank the reviewer for recognizing the value of the session-disjoint, causal, event-level, and recovery-aware evaluation protocol.
Q1: Distinct contribution of ThermalFault ThermalFault provides an evaluation protocol for online thermal motor monitoring. Complete recording sessions are kept separate, and the dataset includes annotated fault onset, fault removal, and recovery periods. This supports causal evaluation of event recall, detection delay, false alarms, post-trigger diagnosis, and recovery behaviour. In contrast to datasets designed mainly for frame-level condition classification, ThermalFault preserves continuous fault and recovery sequences and provides the timestamps required for causal online evaluation. It therefore complements datasets with broader motor or fault diversity by enabling measurements of missed events, detection delay, false alarms, and recovery stability. Three continuous mixed streams are excluded from training and model selection for locked evaluation. These streams expose failure modes that may not be visible from shuffled frame-level evaluation. We will revise the manuscript to position ThermalFault as a controlled and complementary benchmark and to avoid claims of broad cross-motor generalization. Q2: Additional metrics We appreciate the reviewer’s suggestion to provide a more complete quantitative evaluation. Given the limited number of fault events, reporting uncertainty is particularly important. The Stage 1 event recall of 11/12 corresponds to a 95% Wilson confidence interval of approximately 0.65–0.99, indicating that this estimate should be interpreted with care. For frame-level evaluation, accuracy and macro-F1 are reported as the mean and standard deviation across six folds. Classwise F1 is also included to reflect the effect of class imbalance, and we will add per-class accuracy for greater clarity. We will further clarify the definitions and denominators of event recall, detection delay, FAR/h, conditional diagnosis accuracy, and recovery metrics. In particular, FAR/h is computed from alarm rising edges over 5.994 hours of valid normal monitoring, including eligible pre-fault normal segments. We will update the manuscript with per-frame inference time, computational cost, and parameter-sensitivity results measured under the current experimental setup. Evaluating robustness across different motors, camera positions, and ambient conditions requires additional data collection and is beyond the scope of the present study. We will state this limitation explicitly. Future work will extend the benchmark to multiple motors, viewpoints, operating loads, and ambient conditions, and will evaluate deployment efficiency and cross-system robustness under these settings. Q3: Comparison with existing datasets We thank the reviewer for requesting a clearer and more balanced dataset comparison. In the revised manuscript, we have expanded the comparison with existing thermal motor datasets to cover motor and fault diversity, temporal continuity, event annotations, recovery periods, session-disjoint evaluation, and online monitoring metrics. ThermalFault shares with existing datasets the use of infrared images for normal and fault recognition. Its main difference is that it preserves complete temporal sessions and provides fault-onset, fault-removal, and recovery annotations. This design supports causal evaluation of missed events, detection delay, false alarms, post-trigger diagnosis, and recovery stability. ThermalFault was designed as a controlled benchmark around one motor system, rather than as a broad collection of heterogeneous machines. This focused setting enables consistent evaluation of two fault types across the complete monitoring process, including detection, diagnosis, and recovery. It also provides a common testbed for comparing online monitoring methods under the same session-disjoint protocol. However, results from this setting do not establish generalization to other motor systems. We therefore present ThermalFault as a focused benchmark that complements datasets with broader motor, load, and fault diversity.