acceptodds
Under review as a conference paper at ICLR 2027

Reliable Virtual Sensing: A Multi-Domain Benchmark for Robustness Under Sensor Failures

10 views

Abstract

Virtual sensing, the estimation of hard-to-measure quantities from available sensor measurements, is a critical enabler for control and monitoring in cyber-physical systems. However, when sensors fail, learning-based predictors can produce physically implausible estimates that propagate to system-level failures. We argue that real-world deployment demands robustness and introduce MuViS-C, the first multi-domain benchmark of robustness against common sensor failures in learning-based virtual sensing. Building on an existing nominal-performance benchmark and established corruption taxonomies, it covers ten sensor failure modes, from subtle drifts to catastrophic signal dropouts, at multiple severities. These are paired with complementary robustness measures capturing average error under corruption, relative degradation, and worst-case fragility. Across nine datasets from six domains, we benchmark six architectures spanning gradient-boosted trees and the major inductive biases for sequence modeling: convolution, recurrence, attention, and MLP-mixing. On the attention-based architecture, we further probe three robustification strategies. We find that (i) every model degrades substantially under corruption, becoming worse than a naïve predictor on at least one corruption setting, (ii) gradient-boosted tree ensembles achieve strong robustness, and (iii) dedicated robustification closes the gap between the attention-based architecture and the most robust models, though each strategy hurts nominal performance. The benchmark's multi-domain design proves essential, as model rankings shift across datasets, and no single domain captures the full robustness picture. MuViS-C is open-source and extensible to new datasets, failure modes, measures, and models.

Primary area:
datasets and benchmarks
Keywords:
Benchmarking robustness, Deep learning, Virtual sensing, Sensor failure

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_P1jpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2(+1)
Presentation
3
Soundness
1
Read review
Paper summary

The paper introduces MuVis-C, a multi-domain benchmark of robustness under sensor failures for virtual sensing. It evaluates six machine-learning architectures across nine datasets from health, environmental, industrial, energy, and automotive domains by synthetically injecting ten parameterized corruption types, including bias, noise, scaling, drift, outliers, clipping, and stuck-at faults, into one normalized sensor channel at a time. Using measures of average degradation, degradation relative to nominal accuracy, and the severity at which a model becomes worse than a naive predictor, the authors find that every tested model is vulnerable, robustness rankings vary substantially across domains, and good nominal performance does not guarantee robustness. They also compare input dropout, adversarial training, and fault-focused self-supervised pretraining, finding that robustification can help but involves trade-offs and no method performs best universally.

Critical strengths
  1. The framework proposed is systematic and reproducible for comparing model sensitivity to predefined synthetic corruptions across multiple channels and severity levels.
  2. The evaluation covers across nine datasets and six domains. The results show that robustness rankings vary by domain and differ from nominal-performance rankings, cautioning against conclusions based only on clean accuracy or a single dataset.
Critical weaknesses
  1. The paper does not clearly explain the challenges specific to virtual sensing beyond the classical problem of sensor-fault propagation through an estimator, which has been studied extensively in sensor network and CPS literature. Also, the framework only covers isolated single-channel corruptions in open loop, and doesn't include important system behaviors such as correlated failures, communication and timing faults, changing physical conditions, and downstream control effects. So both the novelty and practical significance of the conclusions are limited.

  2. The failure modes used by the framework are from previous literature (Brandt et al., 2025), which, although based on failure taxonomies from earlier work, are also largely synthetically defined rather than derived from real-world failure datasets or validated against real-world fault observations. In sensing applications, one man's noise is another man's signal. The "failures" (noise, drift, etc.) depend on the actual sensing mechanism and application context. Therefore, applying identical normalized corruptions across domains provides limited evidence of real-world sensor-failure robustness. What value such a benchmark offers is unclear.

Questions for authors
  1. What empirical evidence supports the selected corruption functions, parameter ranges, and severity mappings as representative of real sensor failures across the six domains? Were they compared with field-failure traces or domain-specific sensor models?
  2. Which challenges or findings are specific to learning-based virtual sensing, rather than instances of the established problem of sensor-fault propagation through observers, state estimators, or sensor-fusion systems?
  3. How should robustness on isolated, synthetically corrupted channels be interpreted as evidence of deployment reliability, given that real deviations may represent legitimate physical events and that faults can be correlated, context-dependent, or originate in communication and timing?
4 comments

We thank the reviewer for the feedback and are glad that the reviewer finds the framework systematic and its multi-domain results informative beyond clean accuracy.

Q1 and W2 – Evidence for the failure modes, parameter ranges and severities. The failure modes are deliberately common across sensors, and they were not fitted to field-failure traces, which we name as a limitation. Real deployments rarely provide fault ground truth, and their faults are confounded with process changes, so degradation cannot be attributed to a fault mode and severity. We therefore follow established practice in robustness benchmarking and use synthetic corruptions at graded severities, as do ImageNet-C (Hendrycks & Dietterich, 2019), object detection (Michaelis et al., 2019), LiDAR and multi-sensor perception benchmarks with simulated sensor failures (Kong et al., 2023; Beemelmanns et al., 2024), industrial time-series robustness studies (Dix et al., 2023) and an industrial forecasting benchmark (Windmann et al., 2025).

The modes follow two independent sensor-fault taxonomies, one built from the sensing principle (Balaban et al., 2009) and one from the values a sensor reports (Jesus et al., 2017), and recur in deployed sensor networks (Ni et al., 2009; Sharma et al., 2010), soft sensors (Kadlec et al., 2009) and 57 sensor-data-quality studies (Teh et al., 2020). Jesus et al. define faults at the sensor interface, by the deviating values a sensor produces, abstracting from its physical cause. This interface is exactly where a learned virtual sensor observes its inputs and where MuViS-C injects faults, so one shared catalogue applies across sensors and domains (Section 3.2). This is where the value of MuViS-C lies. It gives up sensor-specific detail but makes robustness comparable across sensors and domains, which reveals that rankings shift between domains and differ from clean rankings. Sensor-specific fault models can be added through the failure interface (App. P), as the communication and timing faults in Q3 show.

Because fault magnitudes differ between sensors, we sweep from small to large faults instead of fixing one. A bias or drift ranges from about 3 % of a channel's operating range at s = 0.05 to a median 65 % at s = 1 (3σ, 1st–99th percentile of the training data, 141 channels). We also agree that whether a deviation is a fault depends on the application ("one man's noise is another man's signal"). Legitimate physical events stay in the clean recordings, and because the target keeps its true value while the input is corrupted, any change in the estimate is an error. Telling faults from events at run time is the task of fault detection, which complements the benchmark. With a reliable detector and mean replacement, TST-ISensD barely degrades (rPC 1.03, App. H).

Revision: we will state in Section 3.2 that the corruption parameters are calibrated to each channel's spread rather than to field-failure traces, name this as a limitation in Section 5, and add the per-channel severities in physical units as a new appendix, so that practitioners can locate their own sensors on this scale.

Q2 and W1 – What is specific to learned virtual sensing. The protocol itself is not: it treats a virtual sensor as a mapping from sensor windows to a target, so any estimator can be registered and evaluated, including a model-based one where governing equations are available. What is specific is the need for it. Where the governing equations exist, the sensitivity of an estimate to a faulty sensor can be derived or simulated for the plant at hand. Learned virtual sensors are used where they do not (Kadlec et al., 2009), and for them robustness can only be measured, which requires a fixed catalogue of faults, severity scale, measures and references. Virtual sensing had a benchmark for nominal accuracy but none for robustness.

That measurement cannot be reused across models. Ranking each dataset's channels by their rPC, two architectures trained on the same data and target agree only weakly on which channel is critical (mean pairwise Spearman 0.46, 324 pairs), while retraining the same architecture on another seed reproduces its ranking (0.82, 243 pairs); all nine models share the most critical channel in 1 of 27 dataset × seed combinations. Which failure mode is worst does transfer (0.84). Sensor criticality is thus a property of the trained model and belongs in model selection.

Two further points have no counterpart in the model-based setting, where fault tolerance is organised around the estimator through detection, isolation and reconfiguration. First, a learned virtual sensor can be made robust in itself, through how it is trained or built, and such proposals (adversarial training, input sensor dropout, fault-aware pretraining, architectural changes) can only be credited against a common catalogue of faults, severities, measures and baselines. Our own comparison shows why: the three strategies disagree across measures (ISensD improves rPC, 1.72 → 1.48, but worsens mPC, 1.48 → 1.62) and each trades clean accuracy, so no single number would settle which of them helps. Second, because a learned model is not derived per plant, one model can be evaluated on every domain at once; with time-series and tabular foundation models that estimate a target in context, how one architecture's robustness generalises across domains becomes the natural question, and answering it requires a multi-domain testbed.

Revision: we will add the channel-sensitivity analysis above as a new appendix, state the scope of the protocol and this motivation in Section 1 and Section 3.4, name the evaluation of in-context virtual sensors across domains as future work in Section 5, and add the model-based fault-diagnosis and fault-tolerant estimation literature (Chen & Patton, 1999; Isermann, 2006; Gao et al., 2015; Blanke et al., 2016) to the related work, together with the soft-sensor literature, which names missing values, outliers and drift as practical obstacles (Kadlec et al., 2009).

Q3 and W1 – Interpreting the results as evidence of deployment reliability. MuViS-C is a stress test and provides necessary rather than sufficient evidence. A model that fails under a single moderately faulty channel cannot be expected to cope with compound faults in deployment, while passing the benchmark does not guarantee reliability. Open-loop estimation errors are the input that downstream control has to cope with, and they can lead to closed-loop failures (Brandt et al., 2025). Legitimate physical events stay in the clean recordings (see Q1).

Simultaneous failures are covered in App. L, where the same fault hits up to four randomly chosen channels. The ranking is identical for one to three failing channels and changes only in the middle for four.

To show that new fault modes are easy to add (App. P), we implemented the communication and timing faults the reviewer raises as four modes, each in about ten lines. Delay lags the channel by up to half a window, freeze holds the last value, packet loss drops up to 50 % of samples and holds the last one, and quantisation rounds to steps of up to 1.5σ. We evaluated all models on all nine datasets over the full severity grid (TEP every 5th simulation run).

Model (rPC)delayfreezepacket loss + holdquantisationall fourmean rank
XGBoost1.041.051.001.081.045.0
CatBoost1.041.061.001.121.066.0
M-TCN1.231.401.021.191.239.0
P-TSMixer1.081.191.011.101.104.4
xLSTM-M1.071.141.011.121.096.6
TST1.151.181.021.081.127.2
TST-PGD1.051.051.011.031.032.9
TST-ISensD1.031.061.001.041.032.8
TST-F2F1.101.091.011.061.075.4
Ridge (linear reference)1.311.311.091.111.245.7

These faults are much milder than sensor faults, because they delay or coarsen the signal but keep its values plausible, and only 15 of 90 model–dataset pairs ever become worse than the naive predictor. Their ranking correlates moderately with the main benchmark (Spearman 0.67), which again shows that robustness depends on the fault type. TST-ISensD and TST-PGD are the most robust and M-TCN the least.

Changing operating conditions are part of the data, for example battery drive cycles at five ambient temperatures and three racing sessions with different car–track combinations.

Revision: we will state in Section 5 that the benchmark is a stress test and therefore provides necessary rather than sufficient evidence of deployment reliability, and name common-cause and context-dependent faults and closed-loop effects as the limits of the open-loop scope.

We thank the reviewer again for the questions, which helped us sharpen the scope and the interpretation of the benchmark. If further analyses would be helpful, we are happy to run them.

References

  • Balaban, E., Saxena, A., Bansal, P., Goebel, K. F. & Curran, S. (2009). Modeling, detection, and disambiguation of sensor faults for aerospace applications. IEEE Sensors Journal, 9(12), 1907–1917.
  • Beemelmanns, T., Zhang, Q., Geller, C. & Eckstein, L. (2024). MultiCorrupt: A multi-modal robustness dataset and benchmark of LiDAR-camera fusion for 3D object detection.
  • Blanke, M., Kinnaert, M., Lunze, J. & Staroswiecki, M. (2016). Diagnosis and Fault-Tolerant Control (3rd ed.). Springer.
  • Brandt, J. U., Pütz, N. C., Greiff, M., Lew, T., Subosits, J., Hilbert, M. & Bartz-Beielstein, T. (2025). From faults to features: Pretraining to learn robust representations against sensor failures. NeurIPS 2025.
  • Chen, J. & Patton, R. J. (1999). Robust Model-Based Fault Diagnosis for Dynamic Systems. Kluwer Academic Publishers.
  • Dix, M., Manca, G., Okafor, K. C., Borrison, R., Kirchheim, K., Sharma, D., Chandrika, K. R., Maduskar, D. & Ortmeier, F. (2023). Measuring the robustness of ML models against data quality issues in industrial time series data. IEEE INDIN 2023.
  • Gao, Z., Cecati, C. & Ding, S. X. (2015). A survey of fault diagnosis and fault-tolerant techniques, Part I: Fault diagnosis with model-based and signal-based approaches. IEEE Transactions on Industrial Electronics, 62(6), 3757–3767.
  • Hendrycks, D. & Dietterich, T. (2019). Benchmarking neural network robustness to common corruptions and perturbations. ICLR 2019.
  • Isermann, R. (2006). Fault-Diagnosis Systems. Springer.
  • Jesus, G., Casimiro, A. & Oliveira, A. (2017). A survey on data quality for dependable monitoring in wireless sensor networks. Sensors, 17(9), 2010.
  • Kadlec, P., Gabrys, B. & Strandt, S. (2009). Data-driven soft sensors in the process industry. Computers & Chemical Engineering, 33(4), 795–814.
  • Kong, L., Liu, Y., Li, X., Chen, R., Zhang, W., Ren, J., Pan, L., Chen, K. & Liu, Z. (2023). Robo3D: Towards robust and reliable 3D perception against corruptions. ICCV 2023.
  • Michaelis, C., Mitzkus, B., Geirhos, R., Rusak, E., Bringmann, O., Ecker, A. S., Bethge, M. & Brendel, W. (2019). Benchmarking robustness in object detection: Autonomous driving when winter is coming.
  • Ni, K., Ramanathan, N., Chehade, M. N. H., Balzano, L., Nair, S., Zahedi, S., Kohler, E., Pottie, G., Hansen, M. & Srivastava, M. (2009). Sensor network data fault types. ACM Transactions on Sensor Networks, 5(3), 1–29.
  • Sharma, A. B., Golubchik, L. & Govindan, R. (2010). Sensor faults: Detection methods and prevalence in real-world datasets. ACM Transactions on Sensor Networks, 6(3), 1–39.
  • Teh, H. Y., Kempa-Liehr, A. W. & Wang, K. I.-K. (2020). Sensor data quality: A systematic review. Journal of Big Data, 7(1), 11.
  • Windmann, A., Steude, H., Boschmann, D. & Niggemann, O. (2025). Quantifying robustness: A benchmarking framework for deep learning forecasting in cyber-physical systems. ETFA 2025.
Reviewer_ppt1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces MuViS-C, a robustness benchmark for virtual sensing. It extends the MuViS benchmark (nine datasets from six domains) with ten single-channel failure modes at 20 severity levels. Robustness is summarized by mPC and rPC (adapted from Michaelis et al., 2020) and by a new "baseline crossing severity" scrosss_{\mathrm{cross}}, the lowest severity at which any failure/channel combination makes the RMSE no better than predicting the training mean. Six models are evaluated, and three robustification methods (PGD adversarial training, input sensor dropout and F2F pretraining) are applied to TST. The main findings are that all models fall below the naïve baseline in at least one setting, that tree ensembles are robust without any defense, that robustification places TST in the top rPC cluster at the cost of clean accuracy, and that rankings change across datasets.

Critical strengths
  1. A useful, openly released and well-engineered resource for an under-benchmarked problem: nine datasets, a full failure ×\times channel ×\times severity sweep, an extensible interface and checkpoints. The appendix contains valuable checks, such as three-seed retraining, simultaneous failures of several channels, and per-channel diagnostics.
  2. Concrete and actionable observations, such as tree ensembles being strong robustness baselines and every robustification method trading clean accuracy for robustness. Non-significant results are reported honestly.
Critical weaknesses
  1. The main conclusions rest on metrics that do not measure what is claimed.
    • rPC divides by the clean RMSE and therefore favors models with a worse clean error: on HR, TST-PGD has a worse clean RMSE (13.07 vs. 6.89, Table 8) and a worse mPC than TST, yet a better rPC (1.175 vs. 1.817). On mPC (Table 2), all robustified variants remain behind XGBoost, so "closes the gap" holds mainly for rPC.
    • scrosss_{\mathrm{cross}} is a minimum over a large grid without a confidence interval, often lies at the lowest grid value (0.050, Table 4), and its omnibus test is not significant (p=0.187p=0.187). The claim that every model falls below the naïve predictor thus concerns the single worst cell.
    • The rank tests treat the nine datasets as independent, although three are sessions of the same REVS data and PM10 and PM2.5 share the same source.
  2. The realism and fairness of the study are limited.
    • Drift and time-varying scaling restart in every window (Table 1) instead of accumulating over the stream, and no real failure data are used, so "physically grounded" is overstated.
    • Only PGD is actually searched with the robustness-aware proxy (ISensD has no parameters and F2F is not tuned, Appendix B.5.2), so the gap between robustified TST and the baselines mixes the effect of the method with that of the tuning objective. F2F is pretrained on corruptions from the benchmark's own taxonomy.
    • Most components come from MuViS, F2F and Michaelis et al. (2020), so the novelty lies mainly in scrosss_{\mathrm{cross}} and the severity grid.
Questions for authors
  1. On which metric do you base the claim that robustification "closes the gap" (Weakness 1)? Please support it with the absolute corrupted error or a Pareto plot of clean vs. corrupted error.
  2. What does scrosss_{\mathrm{cross}} show beyond the fact that some cell fails? How do the rank-based conclusions change if the six domains, rather than the nine datasets, are used as the unit of analysis?
  3. Do the conclusions hold if drift and time-varying faults are applied continuously over the test stream? How does F2F perform on failure modes outside its pretraining set, and how much of the gap closes if vanilla TST is tuned with the same robustness-aware proxy as PGD?
  4. All figures are raster images rather than vector graphics. Could you export them as vector graphics (e.g., PDF), in particular the critical-difference diagrams in the appendix (Figs. 9–13), which are low resolution and become blurry when zoomed, enlarge the fonts in Fig. 2 and the severity curves (Figs. 5–8), either show the 95 % CI bands or remove the "95 % CI" legend entry, and rewrite the garbled paragraph on large-language-model usage?

In addition, Listing 1 is merely a README-style usage snippet for the authors’ software package. It conveys no design choice or empirical evidence, yet occupies main-paper space better spent on justifying the benchmark and analyzing results. It should be removed.

4 comments

We thank the reviewer for the detailed review and are glad that the reviewer finds MuViS-C a valuable, well-engineered resource for an under-studied problem.

W1/Q1 – Basis of "closes the gap". The claim is based on rPC, which captures how much a model degrades relative to its own clean error. On rPC, every robustified TST joins the leading cluster (Section 4.2). The reviewer is right that this alone does not support our broader conclusion. As Section 3.3 notes, rPC is meant to be read together with mPC, and because it divides by the clean error, it can favour a model with a worse clean error. On the absolute corrupted error (mPC, Table 2), robustification narrows the gap to the trees but does not close it. The absolute degradation mPC − clean, which involves no division, confirms the rPC result (in your HR example: +2.3 vs. +5.6 RMSE for TST-PGD vs. TST).

ModelcleanmPCabs. degradation (% of naive RMSE)rPC
XGBoost1.451.205.51.28
CatBoost1.301.166.01.38
M-TCN1.031.4015.22.11
P-TSMixer1.891.718.21.39
xLSTM-M1.221.216.31.53
TST1.331.4812.21.72
TST-PGD1.761.435.21.25
TST-ISensD1.691.6210.31.48
TST-F2F1.601.327.61.27
(Clean RMSE and mPC are normalised per dataset to the best of the nine models and averaged with the geometric mean, as in Table 2; 1.0 = best.)
  • On the mPC mean rank, TST-F2F closes 24 % and TST-PGD 12 % of the gap between TST and XGBoost, and TST-ISensD does not improve it.
  • On absolute degradation, TST-PGD degrades less than TST on all 9 datasets and about as little as XGBoost (5.2 vs. 5.5).

We revise the abstract and conclusion accordingly: robustification closes the gap in degradation and narrows it in absolute corrupted error, at a cost in clean accuracy. We add the absolute degradation and a clean-vs.-corrupted plot to the appendix.

W1/Q2 – What s_cross adds. mPC and rPC average over all cells. s_cross adds a worst-case view and shows how early the most vulnerable cell of a model becomes no better than the naive predictor. It is complemented by rPC_worst in App. O, which gives the relative degradation under the single most damaging combination of failure mode and channel instead of the average over all combinations. Because s_cross looks at the worst case, it ranks models differently from the averages (Spearman of mean ranks 0.15 with rPC and 0.32 with mPC). For example, P-TSMixer is 3rd on rPC but joint last on s_cross, while xLSTM-M is 6th on rPC but 1st on s_cross.

Models_cross mean rankrPC mean rank(failure, channel) cells crossing naive (%)
XGBoost5.63.44.6
CatBoost4.94.15.9
M-TCN5.98.118.5
P-TSMixer6.23.813.7
xLSTM-M3.54.75.5
TST6.27.916.1
TST-PGD4.42.77.2
TST-ISensD4.46.219.0
TST-F2F3.74.117.2
  • Not a single cell. Between 4.6 % and 19 % of all (failure, channel) cells cross the naive baseline, depending on the model.
  • Confidence intervals and seeds. Computing s_cross from the lower bootstrap CI bound changes 17 of the 81 values by at most 0.15, and the upper bound changes 21 values by at most 0.30. Across the three training seeds of App. K, the s_cross mean ranks correlate at 0.86 to 0.88, and we add this comparison to the appendix.
  • The 0.05 floor. It mostly reflects sensor loss. Of the 53 hard-fault cells that cross at s ≤ 0.05, 48 already cross at s = 0 (see our response to Reviewer AvAb).

In the revision, we report the share of crossing cells next to s_cross and add the CI-based values to the appendix. We also rephrase the finding that every model becomes worse than the naive predictor, so that it states how many cells cross.

W1/Q2 – Domains as units. The mean ranks barely change when the six domains are the unit of analysis. Spearman between the two rankings is 0.996 for rPC and 0.93 for mPC:

ModelrPC: 9 datasetsrPC: 6 domainsmPC: 9 datasetsmPC: 6 domains
XGBoost3.43.32.32.5
CatBoost4.14.22.62.7
M-TCN8.18.35.65.8
P-TSMixer3.83.57.26.5
xLSTM-M4.74.73.02.7
TST7.97.76.05.8
TST-PGD2.72.85.65.7
TST-ISensD6.26.07.77.5
TST-F2F4.14.55.15.8

We add the domain-level critical-difference diagrams to the appendix. With only six units, the post-hoc test has little power (critical distance 4.9 on a rank scale of 1–9), so the stable mean ranks are the more informative result.

W2/Q3 – Drift accumulating over the stream. MuViS-C is formulated on input windows (Section 3), and we defined drift within each window for two reasons. First, the definition applies to all nine datasets, including those whose test windows do not form a continuous stream. Second, a given severity then means the same fault magnitude in every window, independent of stream length and of where the window lies in the stream. We agree that real drift accumulates over time, so we added stream-level variants of drift and time-varying scaling.

Setup. In the new variants, the fault starts at the beginning of each recorded stream (a drive cycle, racing session or simulation run) and grows until it reaches its full magnitude at the end. Because the test windows slide along the stream, each window sees the drift accumulated so far. Severities are the same as in the main benchmark. We use the six datasets whose test windows form continuous streams (battery, three REVS sessions, vehicle dynamics and TEP), with streams of 89 to 22,148 time steps. Both columns cover linear drift, nonlinear drift and time-varying scaling on these six datasets.

Result. The ranking is preserved (Spearman of the mean ranks 0.98). The two best models (TST-PGD, TST-F2F), XGBoost in 3rd place and M-TCN in last place are unchanged:

ModelrPC per windowrPC over streammean rank: windowmean rank: stream
XGBoost1.401.323.53.7
CatBoost1.561.514.54.8
M-TCN2.442.178.38.3
P-TSMixer1.571.444.34.2
xLSTM-M1.951.686.75.8
TST1.911.727.57.7
TST-PGD1.291.232.02.8
TST-ISensD1.641.385.54.5
TST-F2F1.231.232.73.2

Most rPC values drop because windows early in a stream carry little accumulated drift. The per-window variant applies the full ramp in every window and is therefore the stricter test at equal severity.

W2/Q3 – Tuning objective. ISensD has no hyperparameters, and F2F uses its published settings, so neither benefits from the robustness-aware proxy. To separate PGD's method from its tuning, we tuned vanilla TST with the same proxy, budget and seed. The search covered TST's full architecture and optimiser space, which is larger than PGD's four attack parameters (tuned with the architecture fixed). We then retrained the model and ran the full protocol:

ModelcleanmPCrPCdatasets better than TST (rPC / mPC)
TST1.331.481.72–
TST, robustness-tuned1.541.521.537 / 4
TST-PGD1.761.431.259 / 4

Tuning alone closes 39 % of the rPC gap between TST and TST-PGD, mainly by giving up clean accuracy, and does not improve mPC. Most of PGD's gain therefore comes from adversarial training, not from the tuning objective.

W2/Q3 – F2F outside its pretraining set. We include F2F as one of the few current methods designed specifically for sensor failures, and its authors report that it generalises to unseen fault types. It is pretrained on zeroing, bias and noise, which match three of our ten modes. The other seven never occur in pretraining.

ModelrPC on pretraining modes (hard fault, bias, noise)rPC on the 7 unseen modesmean rank: pretraining modesmean rank: unseen modes
XGBoost1.481.193.73.3
CatBoost1.641.264.84.3
M-TCN2.881.748.38.1
P-TSMixer1.691.263.73.8
xLSTM-M1.791.414.45.0
TST2.231.487.87.6
TST-PGD1.431.172.82.6
TST-ISensD1.801.346.05.7
TST-F2F1.471.183.64.7

F2F removes almost the same share of TST's excess degradation (rPC − 1) on both sets (62 % and 61 %). Its robustness therefore transfers to unseen faults, where it performs at about the level of XGBoost (rPC 1.18 vs. 1.19), although by mean rank it falls from 2nd to 5th there. All values are lower on the unseen modes because these are milder on average.

W2 – Physical grounding and novelty. With "physically grounded" we refer to the fault taxonomies our modes are built on, and we now name this explicitly ("grounded in established sensor-fault taxonomies"). This grounding rests on three points.

  • Every mode instantiates a fault archetype on which two independent taxonomies converge, one built from the sensing principle (Balaban et al., 2009) and one at the sensor interface (Jesus et al., 2017).
  • Each archetype has a known physical cause.
  • Field studies of deployed sensor networks report spikes, noise and stuck values (Sharma et al., 2010).

Controlled injection is a deliberate choice and standard for corruption benchmarks (ImageNet-C; Michaelis et al., 2020). Field fault data are rare, unlabelled and confounded with process changes, so they cannot provide per-mode and per-severity attribution with a known ground truth. We name the lack of field-trace validation as a limitation (see also our reply to Reviewer P1jp, Q1).

Building on validated components was likewise a deliberate choice. MuViS-C combines the MuViS datasets, established fault taxonomies, the corruption-benchmark design of ImageNet-C and the metrics of Michaelis et al. (2020), much as ImageNet-C and Michaelis et al. themselves built on existing datasets and corruptions. We see our contribution in the first standardised robustness protocol for virtual sensing and in the findings it enables. We also expect MuViS-C to gain relevance once time-series foundation models are extended from forecasting to virtual sensing. As their nominal benchmarks saturate, other properties such as robustness become the distinguishing ones, much as ImageNet-C followed the saturation of ImageNet accuracy.

Q4. Thank you for catching these presentation issues. In the revision, we will

  • switch to vector figures with larger fonts and remove the "95 % CI" legend entry,
  • rewrite the paragraph on LLM use,
  • move Listing 1 to the appendix.

We thank the reviewer again for the constructive review, which substantially improved the paper. If any point remains open, we are glad to provide additional analyses.

Reviewer_AvAbOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces MuViS-C, extending MuViS with a standardized sensor-failure evaluation protocol covering ten failure modes across nine datasets from six domains. It compares six models and three robustification methods using mPC, rPC, and baseline crossing severity. The results show varying model rankings across domains, strong tree-ensemble baselines, and trade-offs between nominal performance and relative degradation.

Critical strengths
  1. The benchmark provides a useful, extensible protocol for comparing virtual-sensing models under sensor failures. Its coverage of multiple domains and failure modes, together with varying model rankings across datasets, makes it a valuable evaluation resource.
  2. The evaluation goes beyond aggregate rankings. Complementary metrics and severity curves make the accuracy–robustness trade-off visible, while multiple seeds, simultaneous failures, and channel-level diagnostics help assess stability and reveal localized weaknesses.
Critical weaknesses
  1. The interpretation of s_cross across failure modes needs clarification. For hard faults, severity changes the stuck value, while the original measurements are already fully replaced at the lowest level. A low s_cross therefore need not indicate sensitivity to mild corruption. Appendix H studies mean imputation at s=0, and Appendix O attributes crossings for two models on TEP, but the overall influence of hard faults on s_cross remains unclear. A sensitivity check would help readers interpret this metric.
Questions for authors
  1. Using s_cross, how often do hard faults determine the crossing across models and datasets, beyond the examples in Appendix O?
  2. As a sensitivity check, do the s_cross comparisons change when hard faults are excluded?
2 comments

We thank the reviewer for the feedback and the positive assessment, and we are glad that the reviewer sees MuViS-C as a useful and extensible evaluation resource.

W1 & Q1 + Q2. Hard faults determine the crossing in 60 of the 75 (model, dataset) pairs that cross the naive baseline. On a finer grid (0.01 steps), 48 of the 53 hard-fault (model, dataset, channel) cells that cross at or below s = 0.05 already cross at s = 0, i.e. when the sensor is replaced by its mean (sensor loss, not mild corruption). Excluding hard faults changes the ranking moderately (rank correlation 0.70, TST-ISensD loses most):

Modelcrossing set by hard fault (datasets of 9)of these, crossing already at s = 0s_cross mean ranks_cross mean rank, w/o hard faults
XGBoost925.65.2
CatBoost914.94.8
M-TCN845.95.8
P-TSMixer856.26.1
xLSTM-M703.53.2
TST636.26.1
TST-PGD724.44.1
TST-ISensD204.46.3
TST-F2F413.73.5

Hard faults are not the bottleneck for every model: for TST-ISensD, which is trained with channel dropout, they set the crossing on only 2 of 9 datasets. They are also the easiest fault to catch, because a stuck or lost sensor produces a constant signal that simple anomaly detection flags. Once the channel is then replaced by its mean, TST-ISensD barely degrades (rPC 1.03, App. H). We believe that keeping hard faults in s_cross is worthwhile, because the metric adapts as models improve. For a model that copes with sensor loss, such as TST-ISensD, s_cross already reflects its next-weakest failure mode.

Revision. Following the reviewer's suggestion, we

  • add the attribution table above for all models and datasets to the appendix,
  • add s_cross without hard faults to the appendix as a sensitivity check,
  • revise the description of s_cross in Section 3.3 accordingly and link it to the new appendix analyses.

We thank the reviewer again for this helpful suggestion. If further analyses would be useful, we are happy to run them.

Thank you for the hard-fault analysis and the sensitivity check. They address my concerns. I will maintain my score.

Comments on the paper

1 comment

We thank all reviewers for their constructive and detailed feedback and for the discussion so far. As the discussion period ends today, we remain happy to clarify any open questions. Among other points, we addressed the main comments by:

(1) Clarifying the metrics and claims (ppt1, AvAb). We revise the claim: robustification closes the gap in degradation (confirmed by absolute degradation) and narrows it in absolute corrupted error, at a cost in clean accuracy. For s_cross, we report the share of crossing cells, bootstrap CIs and a sensitivity check without hard faults.

(2) Checking the statistics and fairness of the comparison (ppt1). Rankings are nearly unchanged with domains as the unit of analysis and with drift that accumulates over the stream instead of restarting in every window. A robustness-tuned TST shows that PGD's gain comes mainly from adversarial training.

(3) Positioning the benchmark in the literature (ppt1, P1jp). We extended the related work to model-based fault diagnosis and fault-tolerant estimation, data-driven soft sensors, and further corruption benchmarks in perception, which locates MuViS-C with respect to the established literature on sensor-fault propagation.

(4) Motivating the benchmark design (P1jp). MuViS-C provides the first multi-domain robustness benchmark for learning-based virtual sensing, and its results show why one is needed, since robustness can only be measured and transfers neither between models nor between domains. Following established practice (e.g., ImageNet-C), MuViS-C provides a common catalogue of synthetic faults grounded in sensor-fault taxonomies, since field data rarely provide fault ground truth. It serves as a stress test that provides necessary rather than sufficient evidence of deployment reliability.

Further results and clarifications are in the individual responses.

Edit (19 Nov): The current revision (changes in blue) contains the revised claims and the new analyses (Appendices P–V); the remaining promised changes, such as the formatting issues raised by Reviewer ppt1, will be incorporated in the final version.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.