- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper introduces DCR - a statistical test to detect faults in sensors of a physical system, like an RL-based robot. DCR predicts each sensor measurement and compares to the true measurement. If the deviations are too different from some calibrated thresholds, then the sensor is flagged as faulty. Two key insights: (a) too-small errors may also indicate a feaulty sensor, thus the test is two-sided; (b) detecting which sensor is faulty is a key for remediation. Specifically, a suggested remedy is to replace the faulty sensor input with the predicted sensor.
Critical strengths
- The problem is well-motivated.
- The intuition of the motivation is formalized theoretically.
- The modeling of the statistical method is clearly derived from the motivation.
- Limitations are discussed transparently.
Critical weaknesses
- Novelty is quite marginal. Eventually, the paper applies standard statistical tests to the residuals of a predictor.
- The baseline "PEDM-style NLL, windowed" presents similar results in most tests.
- Writing: While the paper acknowledges more sophisticated baselines, the "temporal-jump detector" takes quite a strong place in the narrative despite being a relatively weak strawman.
- Some relevant prior works are missing: "A scalable algorithm for identifying multiple sensor faults using disentangled RNNs" (Haldimann et al., 2019) addresses a sunukar setup of detecting a faulty sensor; "Detecting Rewards Deterioration in Episodic Reinforcement Learning" (Greenberg & Mannor, ICML 2021) addresses monitoring of episodic RL signals while accounting for temporal dependence and calibrating false alarm rates.
- Abstract length should be reduced ~2x.
Questions for authors
Can you clarify which results measure detection of any faulty sensor, and which results measure correct detection of the specific faulty sensor?