- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies reward-independent runtime monitoring of deployed reinforcement-learning agents using observation–action–next-observation triplets. It introduces an interaction entropy budget and a coupling share defined by normalizing transition mutual information by the sum of the triplet’s marginal entropies. These are accompanied by forward and backward conditional entropies and their difference. The statistics are estimated from discretized sliding windows and calibrated against healthy operation; an alarm is triggered once three of the four readings have crossed their respective thresholds. Experiments with 21 frozen SAC and PPO agents in HalfCheetah evaluate 168 perturbation trials. At the operating point selected within this cohort, the monitor detects 69.0% of perturbations versus 44.0% for windowed reward, with both alarming on 5 of 21 unperturbed controls. The proposed monitor is intended to complement reward by detecting changes in interaction statistics.
Critical strengths
- The paper addresses a practically relevant question: whether changes in an RL agent’s interaction with its environment can be detected even when reward remains relatively insensitive. Monitoring such changes can provide useful information beyond task performance alone. The proposed monitor is straightforward to understand and implement. Mutual information and conditional entropies provide interpretable summaries of dependence and residual uncertainty in the interaction stream, offering a reasonable motivation for using their deviations as monitoring signals.
Critical weaknesses
-
Reward-independent monitoring of RL interactions has been studied through dynamics-based OOD detection, e.g., Haider et al. (2023). While the proposed entropy statistics differ from these methods, the paper does not offer sufficient additional capability or insight. Comparisons with relevant existing monitors are needed to assess this contribution.
-
The experiments use only HalfCheetah and abrupt perturbations, with windowed reward as the only matched external baseline. This leaves unclear whether the findings extend to other environments, gradual changes, or benign operational variation. Also, the theoretical justification for the proposed statistics is also relatively limited.
Questions for authors
-
Since the proposed telemetry is intended to complement reward, could the authors compare reward-only, telemetry-only, and combined monitoring at a matched false-alarm level on held-out controls?
-
How does the proposed method compare with a distributional change detector, such as MMD, or a dynamics-based OOD detector operating on the same interaction stream? A comparison using comparable healthy-reference data and false-alarm calibration would clarify whether the entropy construction offers benefits beyond generic change detection.
-
Beyond the selected operating point and the breakdowns in the appendix, could the authors provide detection–false-alarm curves obtained by varying the thresholds? These results would help assess whether the advantage is stable across operating points.
3 comments
Thank you for the thoughtful response and the additional experiments. The comparisons on the same interaction stream directly address my earlier concern about baselines. My remaining concern is that, at the operating point with 6 of 21 held-out controls raising an alarm, joint-histogram KL matched the aggregate detection rate of the three-of-four telemetry rule. Overall, the new experiments strengthen the paper, although a clear detection advantage over this simple baseline seems to be unestablished.
Response to Reviewer mcXu: what detection parity tells us
Thank you for the follow-up. At 6 of 21 held-out control alarms, joint-histogram KL and fixed three-of-four telemetry both detected 72.6% of perturbations. This establishes a tie in aggregate detection at that operating point. It also raises a separate question about what each measurement tells us about the interaction.
The joint-KL baseline compares the window’s distribution with a healthy reference. Our accounting examines predictive dependence within the window. Mutual information is itself a KL divergence, but of the window’s joint distribution from the product of its own marginals, not from a healthy reference: one measures departure from the past, the other the dependence structure of the present window. The copy and XOR examples in App. B make the distinction concrete. They have the same total coupling share , yet all predictive information is carried by the state term in the copy example and by the conditional action term in the XOR example.
We see the value of retaining those terms separately in the controlled gridworld (§5.1, App. C). The state share fell in all four interventions, while the action share fell by 2.34 and 2.60 baseline standard deviations under action noise and slip and moved by only +0.16 and −0.03 under sensor offset and wind. That pattern is the additional information the accounting contributes alongside the alarm.
Thank you again for a review that clearly improved the paper.
Response to Reviewer mcXu
Thank you for the review. You ask the question every monitoring paper has to answer: what does this offer that existing monitors do not? Your three questions turned out to be the right way to answer it, and they reshaped the paper’s claim.
Against existing monitors on the same stream (W1, Q2, Q3)
We added a one-step dynamics model in the spirit of Haider et al. (an MLP trained on each run’s healthy period, alarming on prediction error), MMD on the same windows, occupancy entropy, and a smoothed histogram KL on the same alphabet. Healthy segments provide the reference statistics and fitted baseline models. Operating points are selected using the other agents’ control and perturbation trials, with the held-out agent excluded from selection. At 6/21 held-out control alarms the telemetry detects 72.6–74.4%, the dynamics model 61.3%, MMD 52.4% and reward 50.6%; the KL test matches the telemetry (72.6%), and KL on the state alone reaches 70.2%. The best of the MMD settings we tested (2,000-step reference, doubled bandwidth) reaches 61.3%. The detection–false-alarm curves you asked for (Q3) are now in App. D.2, with the achieved held-out control alarms at each budget; they cross in places, and the clearest separation is at strict budgets: at 2/21 held-out control alarms the selected 𝑘-of-four rule detects 67.3%, against 38.1% for dynamics prediction error, 38.7% for MMD and 34.5% for reward.
At this operating point, joint-histogram KL and three-of-four telemetry tied in aggregate detection. The accounting gives an exact decomposition of the gap to its ceiling from the recorded triplets and shows how the normalized state and action shares shift. In a grid world with known interventions (§5.1, App. C), the normalised action share 𝐼(𝐴; 𝑆′ ∣ 𝑆)/𝐶 fell by 2.34 and 2.60 baseline SDs under the two randomising perturbations (action noise, slip), and shifted by only +0.16 and −0.03 under the two directionally biased ones (sensor offset, wind).
Reward-only, telemetry-only and combined (Q1)
At 6/21 held-out control alarms each, reward alone detects 50.6%, the selected 𝑘-of-four telemetry 76.8%, and their fixed combination 77.4% (App. D.2): the telemetry carries most of the gain, and the increment from adding reward is small and uncertain. The stronger evidence for complementarity is trial by trial. With sequential (CUSUM) monitors at the same observed alarm count (6/21 nominal runs alarming within 100 episodes each), the telemetry caught 18 faults that reward missed, and reward caught 22 that the telemetry missed — sensing and actuation disturbances on one side, forces acting on the motion on the other. Combining the two sequential monitors raises detection to 78.6–80.4%, from 70.8% for reward and 63–69% for the telemetry, at 8/21 instead of 6/21 runs alarming within 100 episodes.
Gradual changes, benign variation, theory (W2)
In the grid world, gradual faults are tracked as they grow: halfway through the action-noise ramp, 𝑃 had flagged 21 of 21 runs, reward 7 and MMD 4. Under benign changes in operating conditions, the three-offour rule raised the fewest alarms of the readings tested. A new mathematical appendix (App. B) gives the theory: the accounting is defined for any controlled record, with an exact gap identity 𝑃 = 1 2 − [𝐻𝑓 + 𝐻𝑏 + 𝐼(𝑆; 𝐴)]/(2𝐶), the decomposition above, and a precise statement of what the readings can and cannot tell. HalfCheetah remains the only continuous-control environment; we state that scope explicitly and name a second environment as the next step.
Your questions asked us to show two things, and the revision does: the grid world shows what the decomposition reveals under known interventions, and the paired HalfCheetah trials show that reward and telemetry catch different changes.