- Rating
- 2
- Confidence
- 3
- Informativeness
- 2(+1)
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper introduces MuVis-C, a multi-domain benchmark of robustness under sensor failures for virtual sensing. It evaluates six machine-learning architectures across nine datasets from health, environmental, industrial, energy, and automotive domains by synthetically injecting ten parameterized corruption types, including bias, noise, scaling, drift, outliers, clipping, and stuck-at faults, into one normalized sensor channel at a time. Using measures of average degradation, degradation relative to nominal accuracy, and the severity at which a model becomes worse than a naive predictor, the authors find that every tested model is vulnerable, robustness rankings vary substantially across domains, and good nominal performance does not guarantee robustness. They also compare input dropout, adversarial training, and fault-focused self-supervised pretraining, finding that robustification can help but involves trade-offs and no method performs best universally.
Critical strengths
- The framework proposed is systematic and reproducible for comparing model sensitivity to predefined synthetic corruptions across multiple channels and severity levels.
- The evaluation covers across nine datasets and six domains. The results show that robustness rankings vary by domain and differ from nominal-performance rankings, cautioning against conclusions based only on clean accuracy or a single dataset.
Critical weaknesses
-
The paper does not clearly explain the challenges specific to virtual sensing beyond the classical problem of sensor-fault propagation through an estimator, which has been studied extensively in sensor network and CPS literature. Also, the framework only covers isolated single-channel corruptions in open loop, and doesn't include important system behaviors such as correlated failures, communication and timing faults, changing physical conditions, and downstream control effects. So both the novelty and practical significance of the conclusions are limited.
-
The failure modes used by the framework are from previous literature (Brandt et al., 2025), which, although based on failure taxonomies from earlier work, are also largely synthetically defined rather than derived from real-world failure datasets or validated against real-world fault observations. In sensing applications, one man's noise is another man's signal. The "failures" (noise, drift, etc.) depend on the actual sensing mechanism and application context. Therefore, applying identical normalized corruptions across domains provides limited evidence of real-world sensor-failure robustness. What value such a benchmark offers is unclear.
Questions for authors
- What empirical evidence supports the selected corruption functions, parameter ranges, and severity mappings as representative of real sensor failures across the six domains? Were they compared with field-failure traces or domain-specific sensor models?
- Which challenges or findings are specific to learning-based virtual sensing, rather than instances of the established problem of sensor-fault propagation through observers, state estimators, or sensor-fusion systems?
- How should robustness on isolated, synthetically corrupted channels be interpreted as evidence of deployment reliability, given that real deviations may represent legitimate physical events and that faults can be correlated, context-dependent, or originate in communication and timing?
4 comments
Q2 and W1 – What is specific to learned virtual sensing. The protocol itself is not: it treats a virtual sensor as a mapping from sensor windows to a target, so any estimator can be registered and evaluated, including a model-based one where governing equations are available. What is specific is the need for it. Where the governing equations exist, the sensitivity of an estimate to a faulty sensor can be derived or simulated for the plant at hand. Learned virtual sensors are used where they do not (Kadlec et al., 2009), and for them robustness can only be measured, which requires a fixed catalogue of faults, severity scale, measures and references. Virtual sensing had a benchmark for nominal accuracy but none for robustness.
That measurement cannot be reused across models. Ranking each dataset's channels by their rPC, two architectures trained on the same data and target agree only weakly on which channel is critical (mean pairwise Spearman 0.46, 324 pairs), while retraining the same architecture on another seed reproduces its ranking (0.82, 243 pairs); all nine models share the most critical channel in 1 of 27 dataset × seed combinations. Which failure mode is worst does transfer (0.84). Sensor criticality is thus a property of the trained model and belongs in model selection.
Two further points have no counterpart in the model-based setting, where fault tolerance is organised around the estimator through detection, isolation and reconfiguration. First, a learned virtual sensor can be made robust in itself, through how it is trained or built, and such proposals (adversarial training, input sensor dropout, fault-aware pretraining, architectural changes) can only be credited against a common catalogue of faults, severities, measures and baselines. Our own comparison shows why: the three strategies disagree across measures (ISensD improves rPC, 1.72 → 1.48, but worsens mPC, 1.48 → 1.62) and each trades clean accuracy, so no single number would settle which of them helps. Second, because a learned model is not derived per plant, one model can be evaluated on every domain at once; with time-series and tabular foundation models that estimate a target in context, how one architecture's robustness generalises across domains becomes the natural question, and answering it requires a multi-domain testbed.
Revision: we will add the channel-sensitivity analysis above as a new appendix, state the scope of the protocol and this motivation in Section 1 and Section 3.4, name the evaluation of in-context virtual sensors across domains as future work in Section 5, and add the model-based fault-diagnosis and fault-tolerant estimation literature (Chen & Patton, 1999; Isermann, 2006; Gao et al., 2015; Blanke et al., 2016) to the related work, together with the soft-sensor literature, which names missing values, outliers and drift as practical obstacles (Kadlec et al., 2009).
Q3 and W1 – Interpreting the results as evidence of deployment reliability. MuViS-C is a stress test and provides necessary rather than sufficient evidence. A model that fails under a single moderately faulty channel cannot be expected to cope with compound faults in deployment, while passing the benchmark does not guarantee reliability. Open-loop estimation errors are the input that downstream control has to cope with, and they can lead to closed-loop failures (Brandt et al., 2025). Legitimate physical events stay in the clean recordings (see Q1).
Simultaneous failures are covered in App. L, where the same fault hits up to four randomly chosen channels. The ranking is identical for one to three failing channels and changes only in the middle for four.
To show that new fault modes are easy to add (App. P), we implemented the communication and timing faults the reviewer raises as four modes, each in about ten lines. Delay lags the channel by up to half a window, freeze holds the last value, packet loss drops up to 50 % of samples and holds the last one, and quantisation rounds to steps of up to 1.5σ. We evaluated all models on all nine datasets over the full severity grid (TEP every 5th simulation run).
| Model (rPC) | delay | freeze | packet loss + hold | quantisation | all four | mean rank |
|---|---|---|---|---|---|---|
| XGBoost | 1.04 | 1.05 | 1.00 | 1.08 | 1.04 | 5.0 |
| CatBoost | 1.04 | 1.06 | 1.00 | 1.12 | 1.06 | 6.0 |
| M-TCN | 1.23 | 1.40 | 1.02 | 1.19 | 1.23 | 9.0 |
| P-TSMixer | 1.08 | 1.19 | 1.01 | 1.10 | 1.10 | 4.4 |
| xLSTM-M | 1.07 | 1.14 | 1.01 | 1.12 | 1.09 | 6.6 |
| TST | 1.15 | 1.18 | 1.02 | 1.08 | 1.12 | 7.2 |
| TST-PGD | 1.05 | 1.05 | 1.01 | 1.03 | 1.03 | 2.9 |
| TST-ISensD | 1.03 | 1.06 | 1.00 | 1.04 | 1.03 | 2.8 |
| TST-F2F | 1.10 | 1.09 | 1.01 | 1.06 | 1.07 | 5.4 |
| Ridge (linear reference) | 1.31 | 1.31 | 1.09 | 1.11 | 1.24 | 5.7 |
These faults are much milder than sensor faults, because they delay or coarsen the signal but keep its values plausible, and only 15 of 90 model–dataset pairs ever become worse than the naive predictor. Their ranking correlates moderately with the main benchmark (Spearman 0.67), which again shows that robustness depends on the fault type. TST-ISensD and TST-PGD are the most robust and M-TCN the least.
Changing operating conditions are part of the data, for example battery drive cycles at five ambient temperatures and three racing sessions with different car–track combinations.
Revision: we will state in Section 5 that the benchmark is a stress test and therefore provides necessary rather than sufficient evidence of deployment reliability, and name common-cause and context-dependent faults and closed-loop effects as the limits of the open-loop scope.
We thank the reviewer again for the questions, which helped us sharpen the scope and the interpretation of the benchmark. If further analyses would be helpful, we are happy to run them.
References
- Balaban, E., Saxena, A., Bansal, P., Goebel, K. F. & Curran, S. (2009). Modeling, detection, and disambiguation of sensor faults for aerospace applications. IEEE Sensors Journal, 9(12), 1907–1917.
- Beemelmanns, T., Zhang, Q., Geller, C. & Eckstein, L. (2024). MultiCorrupt: A multi-modal robustness dataset and benchmark of LiDAR-camera fusion for 3D object detection.
- Blanke, M., Kinnaert, M., Lunze, J. & Staroswiecki, M. (2016). Diagnosis and Fault-Tolerant Control (3rd ed.). Springer.
- Brandt, J. U., Pütz, N. C., Greiff, M., Lew, T., Subosits, J., Hilbert, M. & Bartz-Beielstein, T. (2025). From faults to features: Pretraining to learn robust representations against sensor failures. NeurIPS 2025.
- Chen, J. & Patton, R. J. (1999). Robust Model-Based Fault Diagnosis for Dynamic Systems. Kluwer Academic Publishers.
- Dix, M., Manca, G., Okafor, K. C., Borrison, R., Kirchheim, K., Sharma, D., Chandrika, K. R., Maduskar, D. & Ortmeier, F. (2023). Measuring the robustness of ML models against data quality issues in industrial time series data. IEEE INDIN 2023.
- Gao, Z., Cecati, C. & Ding, S. X. (2015). A survey of fault diagnosis and fault-tolerant techniques, Part I: Fault diagnosis with model-based and signal-based approaches. IEEE Transactions on Industrial Electronics, 62(6), 3757–3767.
- Hendrycks, D. & Dietterich, T. (2019). Benchmarking neural network robustness to common corruptions and perturbations. ICLR 2019.
- Isermann, R. (2006). Fault-Diagnosis Systems. Springer.
- Jesus, G., Casimiro, A. & Oliveira, A. (2017). A survey on data quality for dependable monitoring in wireless sensor networks. Sensors, 17(9), 2010.
- Kadlec, P., Gabrys, B. & Strandt, S. (2009). Data-driven soft sensors in the process industry. Computers & Chemical Engineering, 33(4), 795–814.
- Kong, L., Liu, Y., Li, X., Chen, R., Zhang, W., Ren, J., Pan, L., Chen, K. & Liu, Z. (2023). Robo3D: Towards robust and reliable 3D perception against corruptions. ICCV 2023.
- Michaelis, C., Mitzkus, B., Geirhos, R., Rusak, E., Bringmann, O., Ecker, A. S., Bethge, M. & Brendel, W. (2019). Benchmarking robustness in object detection: Autonomous driving when winter is coming.
- Ni, K., Ramanathan, N., Chehade, M. N. H., Balzano, L., Nair, S., Zahedi, S., Kohler, E., Pottie, G., Hansen, M. & Srivastava, M. (2009). Sensor network data fault types. ACM Transactions on Sensor Networks, 5(3), 1–29.
- Sharma, A. B., Golubchik, L. & Govindan, R. (2010). Sensor faults: Detection methods and prevalence in real-world datasets. ACM Transactions on Sensor Networks, 6(3), 1–39.
- Teh, H. Y., Kempa-Liehr, A. W. & Wang, K. I.-K. (2020). Sensor data quality: A systematic review. Journal of Big Data, 7(1), 11.
- Windmann, A., Steude, H., Boschmann, D. & Niggemann, O. (2025). Quantifying robustness: A benchmarking framework for deep learning forecasting in cyber-physical systems. ETFA 2025.
We thank the reviewer for the feedback and are glad that the reviewer finds the framework systematic and its multi-domain results informative beyond clean accuracy.
Q1 and W2 – Evidence for the failure modes, parameter ranges and severities. The failure modes are deliberately common across sensors, and they were not fitted to field-failure traces, which we name as a limitation. Real deployments rarely provide fault ground truth, and their faults are confounded with process changes, so degradation cannot be attributed to a fault mode and severity. We therefore follow established practice in robustness benchmarking and use synthetic corruptions at graded severities, as do ImageNet-C (Hendrycks & Dietterich, 2019), object detection (Michaelis et al., 2019), LiDAR and multi-sensor perception benchmarks with simulated sensor failures (Kong et al., 2023; Beemelmanns et al., 2024), industrial time-series robustness studies (Dix et al., 2023) and an industrial forecasting benchmark (Windmann et al., 2025).
The modes follow two independent sensor-fault taxonomies, one built from the sensing principle (Balaban et al., 2009) and one from the values a sensor reports (Jesus et al., 2017), and recur in deployed sensor networks (Ni et al., 2009; Sharma et al., 2010), soft sensors (Kadlec et al., 2009) and 57 sensor-data-quality studies (Teh et al., 2020). Jesus et al. define faults at the sensor interface, by the deviating values a sensor produces, abstracting from its physical cause. This interface is exactly where a learned virtual sensor observes its inputs and where MuViS-C injects faults, so one shared catalogue applies across sensors and domains (Section 3.2). This is where the value of MuViS-C lies. It gives up sensor-specific detail but makes robustness comparable across sensors and domains, which reveals that rankings shift between domains and differ from clean rankings. Sensor-specific fault models can be added through the failure interface (App. P), as the communication and timing faults in Q3 show.
Because fault magnitudes differ between sensors, we sweep from small to large faults instead of fixing one. A bias or drift ranges from about 3 % of a channel's operating range at s = 0.05 to a median 65 % at s = 1 (3σ, 1st–99th percentile of the training data, 141 channels). We also agree that whether a deviation is a fault depends on the application ("one man's noise is another man's signal"). Legitimate physical events stay in the clean recordings, and because the target keeps its true value while the input is corrupted, any change in the estimate is an error. Telling faults from events at run time is the task of fault detection, which complements the benchmark. With a reliable detector and mean replacement, TST-ISensD barely degrades (rPC 1.03, App. H).
Revision: we will state in Section 3.2 that the corruption parameters are calibrated to each channel's spread rather than to field-failure traces, name this as a limitation in Section 5, and add the per-channel severities in physical units as a new appendix, so that practitioners can locate their own sensors on this scale.