acceptodds
Under review as a conference paper at ICLR 2027

ThermalFault: A Session-Disjoint Dataset and Benchmark for Online Motor Fault Monitoring

Abstract

Thermal imaging provides a contact-free method for monitoring motor condition. While thermal images can be evaluated individually and effectively, most applications require capturing the event structure needed for online monitoring and/or the correlation of the adjacent thermal frames. This prior information is useful for assessing false alarms, detection delay, post-detection diagnosis, signal recovery, among others. We introduce ThermalFault, an event-annotated thermal dataset and benchmark for motor fault monitoring. The dataset contains 21,556 frames from 18 development sessions covering normal operation, fan faults, single-phase undervoltage, and motor operation at 40 and 50 Hz. Three held-out mixed sessions with fault and recovery periods are included for external evaluation. ThermalFault defines six tasks: frame classification, normal-only anomaly detection, online event detection, post-trigger fault diagnosis, recovery monitoring, and external stress testing. We examine the proposed dataset using classical, deep-learning, time-series, and pretrained vision baselines. We further propose a two-stage pipeline for online fault detection and diagnosis. The detection stage combines deviation from a learned normal thermal trajectory with temporally smoothed fault probabilities from a supervised image classifier. A causal state machine converts the combined evidence into alarm triggers and classifies each event as either a fan fault or single-phase undervoltage. Across six grouped folds, the pipeline detects 11 of 12 fault events at 0.167 false alarms per normal monitoring hour and correctly diagnoses all 11 detected events. These results support session-disjoint, event-level, and recovery-aware evaluation for online thermal fault monitoring.

Primary area:
datasets and benchmarks
Keywords:
thermal imaging, motor fault diagnosis, condition monitoring, anomaly detection, predictive maintenance

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_fQaPOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces a dataset for evaluating systems that continuously monitor a motor with a thermal camera, detect faults as soon as they occur, and identify the fault type. Specifically, the ThermalFault dataset records fault occurrence and recovery periods at fixed timestamps while the motor operates under three conditions, namely normal operation, cooling-fan failure, and single-phase undervoltage, at two frequencies of 40 Hz and 50 Hz. The development set contains a total of 21,556 frames, comprising 15,556 normal frames, 3,000 cooling-fan-fault frames, and 3,000 undervoltage frames. The dataset supports a range of tasks, including frame classification, anomaly detection, event detection, recovery monitoring, and post-trigger diagnosis. The experiments cover supervised classification, normal-only anomaly detection, and causal event detection and diagnosis, and are conducted with a variety of baselines.

Critical strengths

The paper establishes a benchmark for thermal motor fault monitoring that brings together session-disjoint splits, causal detection, event recall, detection delay, false-alarm rate, and conditional diagnosis accuracy within a single evaluation protocol, enabling multiple tasks to be assessed under common conditions. Furthermore, by incorporating the recovery process itself into the evaluation, the benchmark extends to a task that is not covered by existing thermal motor datasets.

Critical weaknesses
  • The scale and diversity of the dataset appear too limited for a benchmark. Although the dataset contains a large number of frames, they are sampled at 3-second intervals and are therefore highly correlated rather than independent. The effective number of independent observations is thus closer to the number of sessions (18 development and 3 external), which is small for a dataset contribution. The number of fault events is also very limited, and the recording protocol is highly uniform, with every development fault session following the same schedule and a single severity per fault type. This may bias the learned models and leaves many unobserved fault scenarios uncovered. The complexity of the data is also limited, since all sessions are recorded with the same motor, camera, viewpoint, and procedure. In addition, details on the OCR-based temperature recovery are missing, so it is unclear how accurate the OCR is or how closely the recovered values match actual temperatures, which weakens the reliability of the data. Representative thermal images for each condition are not shown, and no statistics such as means, variances, or distributions are reported to characterize the data. As a result, the paper’s contribution as a dataset paper appears weak.

  • The experimental evaluation has several limitations. With only 12 fault events, the sample size is too small, and the statistical reliability is limited by the absence of confidence intervals, hypothesis tests, and standard deviations for the key results. There is no ablation of the proposed Stage 1 detector, so the contribution of each evidence route remains unclear. The practicality of the method is also not assessed, as there is no evaluation of per-frame computational cost, inference speed, or robustness to changes in real-world conditions such as different motors, ambient temperatures, and camera positions. In addition, systematic analysis of the results, such as interpreting failure cases or identifying why undervoltage is difficult to detect, is limited, and the anomaly detection experiments do not include change-point detection or deep learning-based anomaly detection methods. The writing also needs revision, including an incorrect table caption and unclear metric definitions. Moreover, T2 and T3 use the same metrics and differ only in whether the method is trained on normal data alone, which makes T2 closer to a methodological constraint within the same task than a separate task. The paper also lacks analysis of key parameters and of how sensitive the results are to the chosen thresholds and settings. Finally, the effect of class imbalance is not discussed, and per-class accuracy is not reported.

Questions for authors
  • Can the author clearly demonstrate the distinct advantages and effectiveness of the proposed dataset compared to existing datasets?
  • Can the author present all the metrics mentioned in the Critical Weaknesses?
  • Can the author clearly identify the strengths, weaknesses, differences, and similarities by comparing the actual sample with other datasets?
2 comments
AuthorsOpenReview ↗

Response to Reviewer fQaP

We thank the reviewer for recognizing the value of the session-disjoint, causal, event-level, and recovery-aware evaluation protocol.

Q1: Distinct contribution of ThermalFault ThermalFault provides an evaluation protocol for online thermal motor monitoring. Complete recording sessions are kept separate, and the dataset includes annotated fault onset, fault removal, and recovery periods. This supports causal evaluation of event recall, detection delay, false alarms, post-trigger diagnosis, and recovery behaviour. In contrast to datasets designed mainly for frame-level condition classification, ThermalFault preserves continuous fault and recovery sequences and provides the timestamps required for causal online evaluation. It therefore complements datasets with broader motor or fault diversity by enabling measurements of missed events, detection delay, false alarms, and recovery stability. Three continuous mixed streams are excluded from training and model selection for locked evaluation. These streams expose failure modes that may not be visible from shuffled frame-level evaluation. We will revise the manuscript to position ThermalFault as a controlled and complementary benchmark and to avoid claims of broad cross-motor generalization. Q2: Additional metrics We appreciate the reviewer’s suggestion to provide a more complete quantitative evaluation. Given the limited number of fault events, reporting uncertainty is particularly important. The Stage 1 event recall of 11/12 corresponds to a 95% Wilson confidence interval of approximately 0.65–0.99, indicating that this estimate should be interpreted with care. For frame-level evaluation, accuracy and macro-F1 are reported as the mean and standard deviation across six folds. Classwise F1 is also included to reflect the effect of class imbalance, and we will add per-class accuracy for greater clarity. We will further clarify the definitions and denominators of event recall, detection delay, FAR/h, conditional diagnosis accuracy, and recovery metrics. In particular, FAR/h is computed from alarm rising edges over 5.994 hours of valid normal monitoring, including eligible pre-fault normal segments. We will update the manuscript with per-frame inference time, computational cost, and parameter-sensitivity results measured under the current experimental setup. Evaluating robustness across different motors, camera positions, and ambient conditions requires additional data collection and is beyond the scope of the present study. We will state this limitation explicitly. Future work will extend the benchmark to multiple motors, viewpoints, operating loads, and ambient conditions, and will evaluate deployment efficiency and cross-system robustness under these settings. Q3: Comparison with existing datasets We thank the reviewer for requesting a clearer and more balanced dataset comparison. In the revised manuscript, we have expanded the comparison with existing thermal motor datasets to cover motor and fault diversity, temporal continuity, event annotations, recovery periods, session-disjoint evaluation, and online monitoring metrics. ThermalFault shares with existing datasets the use of infrared images for normal and fault recognition. Its main difference is that it preserves complete temporal sessions and provides fault-onset, fault-removal, and recovery annotations. This design supports causal evaluation of missed events, detection delay, false alarms, post-trigger diagnosis, and recovery stability. ThermalFault was designed as a controlled benchmark around one motor system, rather than as a broad collection of heterogeneous machines. This focused setting enables consistent evaluation of two fault types across the complete monitoring process, including detection, diagnosis, and recovery. It also provides a common testbed for comparing online monitoring methods under the same session-disjoint protocol. However, results from this setting do not establish generalization to other motor systems. We therefore present ThermalFault as a focused benchmark that complements datasets with broader motor, load, and fault diversity.

Reviewer_fQaPOpenReview ↗

Official comment by fQaP

Q1: The concerns about Q1 (scale, diversity, number of events, fixed protocols, trained models, OCR relability, and statistical analysis) has not been fully addressed.

Q2 : The analysis of statistical reliability lacks detail, and the ablation is not provided. Furthermore, failure cases and challenges of low-voltage detection also insufficient. These concerns still remain unresolved.

Therefore, I keep my original rating.

Reviewer_XZaDOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces ThermalFault, a thermal imaging dataset and benchmark for online motor fault monitoring. It evaluates frame classification, event detection, diagnosis, and recovery using session-disjoint splits. The authors also propose a two-stage pipeline that combines a thermal residual with image-classification scores for detection, then classifies the fault at the alarm trigger.

Critical strengths
  1. A novel dataset is proposed in this paper.

  2. The session-disjoint split is appropriate for highly correlated thermal frames. Evaluating false alarms, detection delay, and recovery is also more useful for online monitoring than reporting frame accuracy alone.

Critical weaknesses
  1. For the dataset, the scope of generalization is limited. All recordings come from just one motor and one fixed imaging setup. The paper should be more careful about conclusions concerning online use beyond these conditions.

  2. For the method, the novelty is limited. The proposed method is primarily a combination of existing techniques, with potential novelty in how they are integrated for online motor fault monitoring. I would not characterize it as a fundamentally novel machine-learning algorithm.

  3. The experiments are insufficient. Although the paper evaluates a range of classical classifiers, CNN architectures, pretrained visual encoders, and normal-only anomaly-detection methods, the evaluation does not adequately compare the proposed pipeline against existing methods specifically developed for motor fault detection and online monitoring, CUSUM, Page–Hinkley for exsample.

Questions for authors

None

2 comments
AuthorsOpenReview ↗

Response to Reviewer XZaD

We thank the reviewer for recognizing the novelty of the dataset and the value of session-disjoint, event-level evaluation. We address the three concerns below.

  1. Generalization scope. We agree that the current data do not establish generalization across motors or imaging setups. Our objective is to provide a controlled benchmark for studying online thermal motor monitoring, rather than a representative sample of all motors and faults. The independent units are 18 development sessions, including 12 fault sessions, from one motor and test bench. We will explicitly state that the reported performance applies to these conditions. The dataset structure and evaluation protocol are designed to accommodate additional motors, loads, severities, fault types, cameras, and sites, but model transfer to such conditions requires future validation. The three external streams test new event orders and durations, but use the same motor and fault implementations; we will make this limitation explicit.

  2. Method novelty. We agree that the individual components are established techniques and do not claim a fundamentally new learning algorithm. The methodological contribution is their integration into an operational two-stage pipeline that separates causal event triggering from post-trigger diagnosis and evaluates false alarms, delay, diagnosis, and recovery under one protocol. We will revise the wording to describe this as a reference pipeline and emphasize that the primary contributions are the event-annotated dataset, session-disjoint protocol, and recovery-aware benchmark. A new controlled ablation further shows that Stage 1 performance is driven mainly by the supervised DenseNet route: DenseNet-only and the support-gated combination both detected 11/12 events with one false alarm, while residual-only detected 6/12 with 11 false alarms.

  3. Online detection baselines. In response to the reviewer, we added two causal change-detection baselines using thermal-regression residuals. For each fold, the regression model and detector parameters were selected on validation sessions only, and test evaluation used the same event windows, normal-monitoring time, rising-edge definition, and FAR/h calculation as Stage 1. CUSUM detected 5/12 events (fan 4/6; undervoltage 1/6), with 9 false alarms, 1.501 FAR/h, and 289.8-s mean delay. Page-Hinkley detected 6/12 events (fan 6/6; undervoltage 0/6), with 15 false alarms, 2.502 FAR/h, and 259.0-s mean delay. The reference detector achieved 11/12 detections (fan 6/6; undervoltage 5/6), one false alarm, 0.167 FAR/h, and 208.6-s mean delay. These results show that standard residual change detectors remain weak for undervoltage under the common protocol. We will include the complete comparison and selection details in the revision.

Reviewer_XZaDOpenReview ↗

Response to the authors

  1. The comparison between the proposed method and the existing methods shows better performance of the proposed method. It addressed the weakness 3 in my review.
  2. The novelty and the limited generalization of the datasets are my main concerns. So, I would keep my rating of weak reject.
Reviewer_vtAzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents ThermalFault, an event-annotated thermal-image dataset and benchmark for online motor fault monitoring. The development set contains 21,556 frames from 18 sessions and covers normal operation, cooling-fan failure, and single-phase undervoltage at 40 and 50 Hz. Three additional mixed streams are held out for evaluation. Instead of focusing only on frame classification, the benchmark also considers anomaly detection, event detection, fault diagnosis, and recovery monitoring using session-disjoint splits and event-level metrics. The authors also develop a two-stage pipeline that first detects fault events by combining a normal thermal-trajectory residual with DenseNet121 predictions and then diagnoses detected events using DenseNet121–AIMv2 late fusion. The system detects 11 of the 12 development fault events and correctly diagnoses all detected events. However, the held-out evaluation shows that shorter undervoltage events can be missed and that detection and recovery may be slow or unstable. Overall, the paper emphasizes that frame-level accuracy alone does not fully represent the performance of an online monitoring system.

Critical strengths

1.The paper introduces a session-disjoint and event-level benchmark that addresses two important limitations of conventional frame-level evaluation. Separating complete recording sessions reduces the risk of leakage between highly similar adjacent thermal frames, while event-level metrics capture missed events, false alarms, detection delay, diagnosis, and recovery behavior. This provides a more realistic evaluation protocol for online thermal fault monitoring. 2.The benchmark connects several stages of the online monitoring process and includes three fully held-out continuous streams that are excluded from training and model selection. These streams reveal failure modes, including missed undervoltage events, long detection delays, and unstable recovery, that would not be apparent from frame-level classification accuracy alone. This makes the dataset and evaluation framework informative even where the proposed method performs poorly.

Critical weaknesses

1.The evidence is limited by the small number and narrow diversity of independent experiments. Although the dataset contains 21,556 frames, the development set consists of only 18 sessions from a single motor and test bench, with two fault types and two operating frequencies. The main event-level evaluation contains only 12 fault events. Consequently, results such as detecting 11 of 12 events and correctly diagnosing all 11 detected events may be sensitive to individual sessions and provide limited evidence of generalization beyond the experimental setup. 2.The held-out evaluation exposes substantial limitations in the central online-monitoring claim, particularly for undervoltage faults. The system misses the 10-minute and 15-minute undervoltage events, detects the 60-minute undervoltage event only after 1,605 seconds, and produces 16 additional alarm rising edges during recovery before becoming stably inactive after 1,377 seconds. These outcomes raise concerns about whether the proposed system can provide timely and stable monitoring under fault durations not represented by the development protocol.

Questions for authors

1.The event-level results are based on only 12 development fault events. Could the authors report the detection delay and diagnosis outcome for each event and indicate in which session the single false alarm occurred? Please also discuss how stable the overall results are across the six folds. 2.How exactly is FAR/h computed? The denominator implied by one false alarm and a reported FAR/h of 0.167 appears to be approximately six normal monitoring hours, which is consistent with the six normal sessions. Were the normal segments before fault onset in the fault sessions excluded? If so, please explain why, since including these periods would change the reported false-alarm rate. 3.In the held-out streams, the 10-minute and 15-minute undervoltage events were missed, while the 60-minute event was detected only after 1,605 seconds and showed unstable recovery. Could the authors analyze whether these failures are related to event duration, fault severity, operating frequency, event order, or other factors? In particular, S16 was also missed in the development evaluation even though it was not a short-duration event. 4.Section 5.2 is titled “Proposed Pipeline and Ablations,” but I could not find a controlled ablation of the Stage 1 detector. Could the authors compare DenseNet-only, thermal-residual-only, and combined detectors using the same alarm-processing rules, including the EMA, thresholds, and state machine? This would help clarify whether the improvement comes from combining the two evidence sources or from the alarm-processing design.

1 comment
AuthorsOpenReview ↗

Response to Reviewer vtAz

We thank the reviewer for recognizing the value of the session-disjoint, event-level, and recovery-aware evaluation. We agree that the small number of independent sessions limits the evidence and that the external results expose important failure modes.

Scope. Our goal is a controlled benchmark for online thermal motor monitoring, not a representative sample of all motors and faults. The 21,556 correlated frames come from 18 independent development sessions, including 12 fault sessions with one event each; all splits are session-disjoint. We will clarify that the results apply to one motor, test bench, and two fault types. The protocol can accommodate additional motors, loads, severities, faults, and sensing modalities, but transfer to these conditions requires future validation.

Q1: Per-event results. We analyzed all 12 events and will include the full table in revision. Five of six folds detected both test events; Fold 5 detected S14 but missed S16. For the 11 detections, delay was 18-624 s (median 117 s; mean 208.6 s). The sole false alarm occurred during valid pre-fault monitoring in S04. Late fusion diagnosed all 11 detected events correctly; S16 was an end-to-end failure. All fan events were detected (6/6; mean delay 78.5 s), versus 5/6 undervoltage events (364.8 s).

Q2: FAR/h. Pre-fault normal periods were included. FAR/h equals false-alarm rising edges divided by valid normal-monitoring hours; a continuous alarm counts once. Each fold contributes about 40 min from its normal session after the 1,200-s warm-up and 10 min from each of two pre-fault periods. The exact total is 5.994 h, hence 1/5.994 = 0.167 FAR/h. Recovery is excluded because residual thermal effects are evaluated separately using recovery delay, occupancy, and re-triggers.

Q3: Undervoltage failures. Duration matters but is insufficient. Undervoltage showed weaker session-level thermal deviations than cooling loss: 0.356 +/- 0.089 vs. 1.046 +/- 0.398 deg C at 40 Hz, and 0.136 +/- 0.337 vs. 1.574 +/- 0.528 deg C at 50 Hz. M01 (10 min) and M02 (15 min) may have ended before weak, gradual changes reached the alarm condition; M03 required 1,605 s. Yet S16 was missed despite lasting about 25 min: its fault-period middle-region temperature differed from its pre-fault mean by only 0.011 deg C, and neither route triggered. Failures occurred at both frequencies and different event positions, so frequency/order alone are insufficient explanations. With one undervoltage level, severity cannot be separated from session variation. Future work requires multiple levels/durations, direct phase-voltage/current measurements, and broader spatial-temporal evidence.

Q4: Stage 1 ablation. Under the same folds, monitoring time, and causal alarm/event rules, DenseNet-only achieved 11/12 detections, one false alarm, 0.167 FAR/h, and 208.6-s mean delay. Residual-only achieved 6/12, 11 false alarms, 1.835 FAR/h, and 488.5 s. The support-gated combination matched DenseNet-only. A residual alarm was accepted only with DenseNet support, blocking unsupported residual alarms; however, it added no detection or delay improvement because DenseNet already detected the same 11 events. We will clarify that Stage 1 recall came from the supervised route, while the support gate prevented residual evidence from increasing false alarms.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.