acceptodds
Under review as a conference paper at ICLR 2027

Surprise Is Not Enough: Dynamics-Consistent Observation Reliability in Reinforcement Learning

Abstract

Reinforcement learning agents deployed on physical systems act on what their sensors report, and some sensor failures produce readings that still look plausible: a channel freezes at its last value, lags behind the true state, or drifts slowly away from it. A policy cannot distinguish such a reading from a real change in state, and robustness training does not tell it which observation to distrust. We introduce dynamics-consistent reliability (DCR), a per-step, per-channel estimate of whether an observation can be trusted, computed inside the RL loop from the agent's own experience. DCR compares each reading with what a learned dynamics model predicts from past observations and actions, and scores the residual two-sided on the model's own clean scale over several window lengths, the classical innovation test of fault diagnosis, so that a residual that is too small or too biased is itself evidence of a fault. The scoring rule has no fitted parameters and each of its components is standardized against a known null distribution. This design addresses a failure shared by the detectors used in practice, from temporal-jump thresholds and reconstruction error to ensemble disagreement and innovation gating: they score an observation by how surprising it is, and a plausible fault is not surprising; the temporal-jump detector ranks a frozen sensor as more reliable than a working one, with per-channel AUROC of 0.004–0.021 on three of four MuJoCo tasks, and the other surprise scores are weakened. We evaluate DCR on nine sensor-fault families with per-step ground truth, in closed loop, with every family held out of tuning when tested. Over five seeds DCR reaches mean held-out AUROC of 0.994 on Hopper, 0.986 on Walker2d, 0.995 on HalfCheetah and 0.792 on Ant, and outperforms PEDM- and DEXTER-style RL out-of-distribution detectors in every task; averaging the PEDM-style score over a window recovers most of that margin, so temporal accumulation is the larger lever and two-sided scoring adds a consistent 0.1–3.6 points. A learned head trained on injected faults is below the fixed statistic on average over held-out families. We further characterize when history-only estimation is impossible, derive a single-channel drift-detectability bound and show that the other channels carry most of the evidence, and build a controller that dead-reckons a flagged channel through the dynamics model and recovers 47–65% of the return lost to a fault on HalfCheetah, while failing on Walker2d, where isolation errs, and on two-channel faults. Code, fault definitions, trained policies and results are released.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, observation corruption, reliability estimation, fault detection, robustness, uncertainty estimation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_HrVJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper introduces DCR - a statistical test to detect faults in sensors of a physical system, like an RL-based robot. DCR predicts each sensor measurement and compares to the true measurement. If the deviations are too different from some calibrated thresholds, then the sensor is flagged as faulty. Two key insights: (a) too-small errors may also indicate a feaulty sensor, thus the test is two-sided; (b) detecting which sensor is faulty is a key for remediation. Specifically, a suggested remedy is to replace the faulty sensor input with the predicted sensor.

Critical strengths
  • The problem is well-motivated.
  • The intuition of the motivation is formalized theoretically.
  • The modeling of the statistical method is clearly derived from the motivation.
  • Limitations are discussed transparently.
Critical weaknesses
  • Novelty is quite marginal. Eventually, the paper applies standard statistical tests to the residuals of a predictor.
  • The baseline "PEDM-style NLL, windowed" presents similar results in most tests.
  • Writing: While the paper acknowledges more sophisticated baselines, the "temporal-jump detector" takes quite a strong place in the narrative despite being a relatively weak strawman.
  • Some relevant prior works are missing: "A scalable algorithm for identifying multiple sensor faults using disentangled RNNs" (Haldimann et al., 2019) addresses a sunukar setup of detecting a faulty sensor; "Detecting Rewards Deterioration in Episodic Reinforcement Learning" (Greenberg & Mannor, ICML 2021) addresses monitoring of episodic RL signals while accounting for temporal dependence and calibrating false alarm rates.
  • Abstract length should be reduced ~2x.
Questions for authors

Can you clarify which results measure detection of any faulty sensor, and which results measure correct detection of the specific faulty sensor?

Reviewer_44fPOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduces Dynamics-Consistent Reliability (DCR), which detects sensor faults in RL using learned dynamics and two-sided prediction error tests. It provides some theoretical analysis and evaluates DCR across four MuJoCo tasks and nine fault types, showing improved detection over existing methods and partial recovery of control performance.

Critical strengths
  1. The paper addresses an important problem of identifying unreliable observation channels rather than merely maintaining policy performance under corrupted observations.
Critical weaknesses
  1. This paper uses extremely heavy use of LLMs to the extent where it makes reading the paper almost impossible. Despite this, I still read the paper once. This is a major weakness which I believe cannot be addressed during the rebuttals -- and based on this reason alone, I am voting for a rejection of this paper. The authors should consider an entire re-write of the paper, with very controlled LLM usage to refine sentences to enhance them for clarity.

What makes me believe this has heavy LLM usage? To name a few examples:

  • "Code, fault definitions and the full breakdown are released."
  • "Two findings run against our initial design"
  • "WHEN SURPRISE CANNOT WORK"
  • "The fix is two-sided."
  1. The authors state:

To our knowledge no prior work evaluates which observation the agent should not trust with per-step, per-channel ground truth, fault families held out of training, an estimator fit inside the RL loop, closed-loop evaluation, and a two-sided test.

This novelty claim is extremely narrow, relying on five specific conditions being met simultaneously. Prior work, including RAPT (Munn et al., 2026) -- that the authors have cited, and classical fault diagnosis -- already explores per-channel monitoring and sensor fault detection. The authors should clarify what is methodologically new beyond combining existing techniques in a new evaluation setting.

  1. The theoretical contributions are limited. Proposition 1 states that detectors looking for large prediction errors can miss faults that produce smaller errors. Proposition 2 states that faults cannot be detected if they produce observations identical to normal behavior. Both are well-known ideas, and neither explains why DCR should outperform existing methods. The authors should clarify what is theoretically insightful here.

  2. Since there are no real-world experiments, it's hard to reconcile if the synthetic faults injected are realistic in nature. The evaluation relies entirely on synthetic, mostly single-channel, persistent faults and clean training data. In the case of real sensors there could be intermittent failures, correlated faults, and measurement noise. So, the authors should either evaluate DCR under more realistic fault patterns + imperfect dynamics models or directly on real hardware (which I understand is not possible for all labs - since you'd need a robot).

Questions for authors
  1. How would DCR work for high-dimensional observations such as images? All the expts. shown are for low-level state observations. The definition of "channel" would chance in that scenario. Further, calibration would also be very non-trivial.
Reviewer_1ERmOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies per-step, per-channel detection of sensor faults in reinforcement-learning systems. It argues that common surprise-based detectors can fail on plausible faults such as frozen, lagged, or slowly drifting sensors because these faults may produce residuals that are smaller than clean residuals.

The paper proposes Dynamics-Consistent Reliability (DCR). DCR uses a learned one-step dynamics model, normalises prediction residuals by the model’s estimated uncertainty, and applies two-sided tests to the residual mean and variance over several window lengths. The method combines classical innovation testing with a learned dynamics model inside the RL loop.

The paper validates DCR on four MuJoCo tasks, nine synthetic fault families, two training checkpoints, and five seeds. It compares DCR with temporal differences, reconstruction error, PEDM-style scores, a DEXTER-style detector, CUSUM, a chi-squared innovation test, and a learned classification head. DCR performs strongly on Hopper, Walker2d, and HalfCheetah, but substantially worse on Ant. A reliability-aware controller recovers part of the lost return on HalfCheetah and Hopper, but fails on several Walker2d cases and on multi-channel faults.

Critical strengths

The paper identifies a concrete failure of standard surprise-based detection: Proposition 1 and the frozen-sensor experiments show that a detector can rank faulty observations as more reliable than clean observations. This is stronger and more useful than showing only a small loss in average detection accuracy. The per-channel evaluation makes the failure visible instead of allowing closed-loop changes in other channels to hide it.

The experimental design isolates the important components well: The paper compares one-sided and two-sided tests, per-step and windowed scores, single-scale and multi-scale scores, learned and fixed scores, and full-input and leave-channel-out predictors. It also includes held-out fault-family evaluation, closed-loop testing, detection delay, false alarms, and downstream control performance. These analyses support the main conclusion that temporal accumulation is the largest factor, while two-sided scoring provides an additional gain.

Critical weaknesses

The statistical calibration claim is stronger than the evidence supports: The paper treats the windowed mean and variance statistics as standard normal under the null. This requires assumptions about residual calibration, independence, temporal correlation, and distributional shape that are not established for the learned heteroscedastic predictor. In particular, the variance statistic is only approximately normal under restrictive conditions. Empirical quantile calibration partly avoids this problem, but then the “known null distribution” is less central than the paper claims.

The evaluation does not yet establish reliability outside synthetic simulation: All faults are injected into MuJoCo observations. The clean residuals are small, and the authors explicitly state that hardware residuals may be larger and less well calibrated. In addition, thresholds are calibrated on the same clean episodes used to report false-alarm rates. The weak Ant results and the Walker2d controller failures show that performance depends strongly on observability, predictor quality, and threshold transfer. This limits the current deployment claims.

Questions for authors

Clarification affecting soundness: What empirical evidence shows that the normalised residuals are sufficiently Gaussian, temporally independent, and correctly scaled for the stated null distributions? Please provide residual calibration diagnostics, autocorrelation results, and performance after calibrating thresholds on a separate clean split.

Clarification affecting novelty and significance: Relative to classical two-sided innovation tests, what is the precise methodological novelty of DCR beyond applying the test to a learned RL dynamics model, using several fixed windows, and taking a maximum? Please identify which part should be viewed as the new algorithmic contribution rather than a new evaluation setting and application.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.