acceptodds
Under review as a conference paper at ICLR 2027

AM I STILL COUPLED TO MY ENVIRONMENT? TASK-AGNOSTIC RUNTIME TELEMETRY FROM THE REINFORCEMENT-LEARNING INTERACTION LOOP

Abstract

After deployment, reward reports whether a reinforcement-learning agent is achieving its designed objective, but not necessarily whether the interaction producing that outcome remains characteristic of healthy operation. As a taskspecific scalar projection, reward can map different agent–environment interaction regimes to similar outcomes. We therefore ask whether the standard observation, action and next-observation triplet, (S, A, S′), can provide a complementary, reward-independent runtime reading. We introduce the interaction entropy budget C = H(S)+H(A)+H(S′) and its coupling share P = I(S,A; S′)/C as a firstperson, reward-independent reading of the realised MDP transition. This is an accounting view of standard Shannon quantities, not a new MDP or informationtheoretic formalism. Combined with the residual-uncertainty companions Hf , Hb and ΔH, it yields an individually calibrated runtime monitor that uses the current interaction stream under a fixed representation, without reward, a predefined fault model or policy internals. Across 21 frozen SAC and PPO agents in HalfCheetah, the monitor detected 69.0% of 168 perturbation trials, compared with 44.0% for matched windowed reward; the agent-clustered advantage was 25.0 percentage points (95% interval [14.9,35.1]), and each alarmed on 5 of 21 unperturbed controls. In a within-cohort ablation at the same five-control-alarm count, the budget normalisation detects 9 points more than raw mutual information and 18 more than the tested min-entropy normalisation. Across these perturbations, the monitor detects interaction-regime changes that the task-outcome projection can miss. It complements rather than replaces reward: it asks whether the agent is still interacting with its environment in the characteristic way established during healthy operation. Such telemetry could flag interaction drift or partial decoupling and help interpret whether a reward change accompanies a broader change in the closed loop. For deployed autonomous systems, the results show why tracking task outcomes alone can be insufficient for runtime monitoring: operators may also need to verify that the underlying control loop remains characteristic of healthy operation.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, runtime monitoring, deployment monitoring, information theory, mutual information, distribution shift, anomaly detection, agent-environment interaction, continuous control, reward-free evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_mcXuOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies reward-independent runtime monitoring of deployed reinforcement-learning agents using observation–action–next-observation triplets. It introduces an interaction entropy budget and a coupling share defined by normalizing transition mutual information by the sum of the triplet’s marginal entropies. These are accompanied by forward and backward conditional entropies and their difference. The statistics are estimated from discretized sliding windows and calibrated against healthy operation; an alarm is triggered once three of the four readings have crossed their respective thresholds. Experiments with 21 frozen SAC and PPO agents in HalfCheetah evaluate 168 perturbation trials. At the operating point selected within this cohort, the monitor detects 69.0% of perturbations versus 44.0% for windowed reward, with both alarming on 5 of 21 unperturbed controls. The proposed monitor is intended to complement reward by detecting changes in interaction statistics.

Critical strengths
  1. The paper addresses a practically relevant question: whether changes in an RL agent’s interaction with its environment can be detected even when reward remains relatively insensitive. Monitoring such changes can provide useful information beyond task performance alone. The proposed monitor is straightforward to understand and implement. Mutual information and conditional entropies provide interpretable summaries of dependence and residual uncertainty in the interaction stream, offering a reasonable motivation for using their deviations as monitoring signals.
Critical weaknesses
  1. Reward-independent monitoring of RL interactions has been studied through dynamics-based OOD detection, e.g., Haider et al. (2023). While the proposed entropy statistics differ from these methods, the paper does not offer sufficient additional capability or insight. Comparisons with relevant existing monitors are needed to assess this contribution.

  2. The experiments use only HalfCheetah and abrupt perturbations, with windowed reward as the only matched external baseline. This leaves unclear whether the findings extend to other environments, gradual changes, or benign operational variation. Also, the theoretical justification for the proposed statistics is also relatively limited.

Questions for authors
  1. Since the proposed telemetry is intended to complement reward, could the authors compare reward-only, telemetry-only, and combined monitoring at a matched false-alarm level on held-out controls?

  2. How does the proposed method compare with a distributional change detector, such as MMD, or a dynamics-based OOD detector operating on the same interaction stream? A comparison using comparable healthy-reference data and false-alarm calibration would clarify whether the entropy construction offers benefits beyond generic change detection.

  3. Beyond the selected operating point and the breakdowns in the appendix, could the authors provide detection–false-alarm curves obtained by varying the thresholds? These results would help assess whether the advantage is stable across operating points.

3 comments
AuthorsOpenReview ↗

Response to Reviewer mcXu

Thank you for the review. You ask the question every monitoring paper has to answer: what does this offer that existing monitors do not? Your three questions turned out to be the right way to answer it, and they reshaped the paper’s claim.

Against existing monitors on the same stream (W1, Q2, Q3)

We added a one-step dynamics model in the spirit of Haider et al. (an MLP trained on each run’s healthy period, alarming on prediction error), MMD on the same windows, occupancy entropy, and a smoothed histogram KL on the same alphabet. Healthy segments provide the reference statistics and fitted baseline models. Operating points are selected using the other agents’ control and perturbation trials, with the held-out agent excluded from selection. At 6/21 held-out control alarms the telemetry detects 72.6–74.4%, the dynamics model 61.3%, MMD 52.4% and reward 50.6%; the KL test matches the telemetry (72.6%), and KL on the state alone reaches 70.2%. The best of the MMD settings we tested (2,000-step reference, doubled bandwidth) reaches 61.3%. The detection–false-alarm curves you asked for (Q3) are now in App. D.2, with the achieved held-out control alarms at each budget; they cross in places, and the clearest separation is at strict budgets: at 2/21 held-out control alarms the selected 𝑘-of-four rule detects 67.3%, against 38.1% for dynamics prediction error, 38.7% for MMD and 34.5% for reward.

At this operating point, joint-histogram KL and three-of-four telemetry tied in aggregate detection. The accounting gives an exact decomposition of the gap to its ceiling from the recorded triplets and shows how the normalized state and action shares shift. In a grid world with known interventions (§5.1, App. C), the normalised action share 𝐼(𝐴; 𝑆′ ∣ 𝑆)/𝐶 fell by 2.34 and 2.60 baseline SDs under the two randomising perturbations (action noise, slip), and shifted by only +0.16 and −0.03 under the two directionally biased ones (sensor offset, wind).

Reward-only, telemetry-only and combined (Q1)

At 6/21 held-out control alarms each, reward alone detects 50.6%, the selected 𝑘-of-four telemetry 76.8%, and their fixed combination 77.4% (App. D.2): the telemetry carries most of the gain, and the increment from adding reward is small and uncertain. The stronger evidence for complementarity is trial by trial. With sequential (CUSUM) monitors at the same observed alarm count (6/21 nominal runs alarming within 100 episodes each), the telemetry caught 18 faults that reward missed, and reward caught 22 that the telemetry missed — sensing and actuation disturbances on one side, forces acting on the motion on the other. Combining the two sequential monitors raises detection to 78.6–80.4%, from 70.8% for reward and 63–69% for the telemetry, at 8/21 instead of 6/21 runs alarming within 100 episodes.

Gradual changes, benign variation, theory (W2)

In the grid world, gradual faults are tracked as they grow: halfway through the action-noise ramp, 𝑃 had flagged 21 of 21 runs, reward 7 and MMD 4. Under benign changes in operating conditions, the three-offour rule raised the fewest alarms of the readings tested. A new mathematical appendix (App. B) gives the theory: the accounting is defined for any controlled record, with an exact gap identity 𝑃 = 1 2 − [𝐻𝑓 + 𝐻𝑏 + 𝐼(𝑆; 𝐴)]/(2𝐶), the decomposition above, and a precise statement of what the readings can and cannot tell. HalfCheetah remains the only continuous-control environment; we state that scope explicitly and name a second environment as the next step.

Your questions asked us to show two things, and the revision does: the grid world shows what the decomposition reveals under known interventions, and the paired HalfCheetah trials show that reward and telemetry catch different changes.

Thank you for the thoughtful response and the additional experiments. The comparisons on the same interaction stream directly address my earlier concern about baselines. My remaining concern is that, at the operating point with 6 of 21 held-out controls raising an alarm, joint-histogram KL matched the aggregate detection rate of the three-of-four telemetry rule. Overall, the new experiments strengthen the paper, although a clear detection advantage over this simple baseline seems to be unestablished.

AuthorsOpenReview ↗

Response to Reviewer mcXu: what detection parity tells us

Thank you for the follow-up. At 6 of 21 held-out control alarms, joint-histogram KL and fixed three-of-four telemetry both detected 72.6% of perturbations. This establishes a tie in aggregate detection at that operating point. It also raises a separate question about what each measurement tells us about the interaction.

The joint-KL baseline compares the window’s distribution with a healthy reference. Our accounting examines predictive dependence within the window. Mutual information is itself a KL divergence, but of the window’s joint distribution from the product of its own marginals, not from a healthy reference: one measures departure from the past, the other the dependence structure of the present window. The copy and XOR examples in App. B make the distinction concrete. They have the same total coupling share PP, yet all predictive information is carried by the state term in the copy example and by the conditional action term in the XOR example.

We see the value of retaining those terms separately in the controlled gridworld (§5.1, App. C). The state share fell in all four interventions, while the action share fell by 2.34 and 2.60 baseline standard deviations under action noise and slip and moved by only +0.16 and −0.03 under sensor offset and wind. That pattern is the additional information the accounting contributes alongside the alarm.

Thank you again for a review that clearly improved the paper.

Reviewer_Dp9rOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a reward-free runtime monitor for deployed RL agents. Since reward can map different interaction regimes to the same outcome, the authors instead track the transition triplet (S, A, S’) through an entropy budget C = H(S) + H(A) + H(S’), a coupling share P = I(S, A; S’)/C, and three residual-uncertainty statistics (H_f, H_b, ΔH), all calibrated per agent on its own healthy period and computed without reward, a fault model or policy internals. On 21 frozen SAC and PPO agents in HalfCheetah, the monitor detects 69.0% of 168 perturbations versus 44.0% for windowed reward (advantage 25.0 points, 95% interval [14.9, 35.1]), with both alarming on 5 of 21 controls; the budget normalisation beats raw mutual information by 9 points and min-entropy normalisation by 18. The monitor is meant to complement reward by checking whether the agent still interacts with its environment as it did when healthy.

Critical strengths

Framing coupling as something the agent can assess from the inside, with no reward signal, is interesting and potentially quite useful. The paper distinguishes it from both input drift and reward monitoring.

The protocol is careful: paired perturbation and control trials, per-agent calibration, agent-clustered bootstrap intervals, the same marking rule for the reward baseline—and the writing is candid.

Critical weaknesses

The main weakness is scope: everything is shown on one environment, with one family of perturbations, one bin count and one window size. Every claim below is therefore a claim about that single setting, and none of the paper’s conclusions (that the statistics detect drift, that the three-of-four rule is a usable operating point, that the entropy quantities carry information beyond occupancy) can be separated from choices specific to it. A second domain with a different state geometry and perturbation type, plus sensitivity to bin count and window size, is the minimum needed before any of the following can be read as general.

No drift-detection baseline. Given a window of (S, A, S′) tuples, the obvious comparison is a two-sample or change-point test on the stream (MMD or any standard detector); the authors say this is the baseline they would add first. Without it there is no evidence the entropy statistics add anything beyond noticing that the set of occupied bins has changed.

Operating point selected on the test data. The three-of-four rule was chosen on the same 21 controls that yield the 5/21 false-alarm rate. Leave-one-agent-out (Appendix B) picks k = 2 in 20 of 21 folds and gives 11/21 control alarms, i.e. roughly half of healthy agents. That is the trustworthy number and belongs in the abstract.

How would a healthy agent’s alarm grow with run length. Especially longer horizons.

The information-theoretic layer is likely a binning artifact. With three bins per coordinate and a deterministic policy, ΔH ≈ −H(A | S) follows from discretization, and the ~2.45-bit jump in C under 3% observation noise is consistent with counting newly occupied cells. The bias-corrected estimator the authors mention as optional is required to settle this.

Questions for authors

Q1. Can you show results for additional tasks and domains.

Q2. What ia the effect of run length?

Q3. Can you clarify the information theoretic derivation?

Q4. Can you convert reward-bases scenes for this problem using something like log likelihood?

3 comments
AuthorsOpenReview ↗

Response to Reviewer Dp9r

Thank you for the review, for seeing the point of the paper — a reading of the loop that the agent can take from the inside, with no reward signal — and for pressing on scope. Our revision turns on one distinction your review made us draw sharply: the accounting identities hold exactly for a declared discrete record, while the readings and alarms depend on how that record is represented, estimated and monitored. We state the first and test the second.

What is exact, and what depends on representation (Q3, W5)

The new mathematical appendix (App. B) gives the identities for the discrete distribution of any controlled record (𝑆, 𝐴, 𝑆′): the decomposition 𝐼(𝑆, 𝐴; 𝑆′) = 𝐼(𝑆; 𝑆′) + 𝐼(𝐴; 𝑆′ ∣ 𝑆), the gap identity 𝑃 = 12 − [𝐻𝑓 +𝐻𝑏 +𝐼(𝑆; 𝐴)]/(2𝐶), and Δ𝐻 = [𝐻(𝑆′)−𝐻(𝑆)]−𝐻(𝐴 ∣ 𝑆). The last reduces to −𝐻(𝐴 ∣ 𝑆) when the two marginal entropies agree; for plug-in histograms of a closed-loop window we bound their difference explicitly (it is 𝑂(log 𝑊 /𝑊 ) with a bounded number of boundary mismatches per window). Your reading of Δ𝐻 is therefore correct: under a deterministic policy on continuous states, 𝐻(𝐴 ∣ 𝑆) is a property of the binning, and we now call Δ𝐻 a representation-dependent diagnostic. 𝑃 depends on the representation too. What changes with coverage-adjusted estimation is its behaviour under faults: mean 𝑃 fell in all eight conditions, in 19–21 of 21 agents for the four strongest.

Beyond one setting (W1, Q1)

A second setting we could add in the discussion period had to be one where we can hold visitation fixed while changing the transition dynamics — which logged rollouts do not allow. We built a grid world with exact population quantities (§5.1, App. C) and sampled it at HalfCheetah’s coverage. With visitation held fixed, a change of dynamics moved 𝑃 by −5.42 healthy-window SDs, while 𝐻(𝑆) moved by −0.08 and the MMD scores moved little. The grid world also covers all four combinations of agent- vs environment-side and randomising vs directionally biased perturbations, gradual faults, and benign changes of operating conditions. In HalfCheetah we varied bins and windows (3–5 bins, 𝑊 = 250–1000): the selected 𝑘-of-four telemetry rule (distinct from the fixed three-of-four rule) detected more than reward in all nine settings (+11.9 to +26.2 points), with achieved held-out control alarms varying from 5 to 11 of 21 and the interval excluding zero in five. A second continuous-control environment remains the most important next step, and we say so. Baselines and the operating point (W2, W3)

Thresholds are now selected on the other agents and evaluated on the held-out agent: three-of-four 72.6%, 𝑃 74.4%, reward 50.6%, all at 6/21 held-out control alarms. On the same stream, MMD reaches 52.4% (61.3% at the best of the settings we tested) and a dynamics model 61.3%; a smoothed histogram KL on the joint alphabet ties three-of-four at 72.6%, and KL on the state alone reaches 70.2%. That tie establishes comparable performance at this operating point. What the information components add is shown elsewhere — by the grid world and by the decomposition into state and action terms, which say which measured dependence in the loop changed.

A stronger reward monitor (Q4)

We added a two-sided CUSUM on standardised per-episode reward (§5.3, App. D.6), with the same held-out selection, and applied the same detector to the telemetry. At a matched long-run target (6/21 runs alarming within 100 nominal episodes each), the telemetry detects 68.5% and reward 70.8%. The two catch different faults: the telemetry caught 18 trials that reward missed, and reward caught 22 that the telemetry missed — sensing and actuation disturbances on one side, forces acting on the motion on the other. Combined, they reach 78.6–80.4%, at a higher 8/21 runs alarming within 100 episodes. This is the outcome/process distinction of the paper, now measured trial by trial.

Run length (Q2)

False alarms accumulate with exposure (§5.3, App. D.5). We re-ran every agent for 1,000 nominal episodes: with the original latched rules calibrated to a 25% target, 6/21 runs (28.6%) alarmed by 100 episodes, rising to 57–67% by 985 episodes, at a similar rate for the telemetry and reward. At that calibration the telemetry detects 46.4% of the archived faults and windowed reward 31.0% (interval including zero). Sequential detection, as in Q4, is a trade-off: CUSUM gave higher fault detection in these comparisons, but 81–86% of nominal runs had alarmed by episode 985, against 57–67% for the latched rules.

Your review pushed us from one result to a set of measured properties, and the paper is better for it.

Thank you for your response and efforts. My only remaining concern which I realize cannot be answered in the limited discussion time is that much more experimentation would make the argument more convincing.

AuthorsOpenReview ↗

Response to Reviewer Dp9r: what further experiments would test

Thank you for engaging with our response, and for acknowledging the time constraint. We agree that more environments would strengthen the practical picture, and we plan them. We would add one clarification on what they would test. The paper makes two kinds of claims. The accounting identities (the decomposition, the gap identity and the bound) are exact for any discrete record; additional environments would illustrate them but cannot change them. The empirical claim, that the readings respond to a change of transition dynamics at fixed visitation, was tested where it could fail: in a setting with known ground truth, where the readings moved as predicted and occupancy-based measures did not. HalfCheetah then shows the same accounting at the scale of continuous control. Broader experiments would chiefly establish how detection rates generalise, which we see as the natural next step rather than a precondition for the accounting itself. Thank you again for a review that shaped the revision.

Reviewer_iTwqOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a reward-free runtime monitor for a frozen RL policy built from sliding-window plug-in Shannon statistics of the discretised (S,A,S′)(S,A,S') stream. The central object is the "interaction entropy budget" C=H(S)+H(A)+H(S′)C = H(S)+H(A)+H(S') and its "coupling share" P=I(S,A;S′)/CP = I(S,A;S')/C, accompanied by Hf=H(S′∣S,A)H_f = H(S'\mid S,A), Hb=H(S,A∣S′)H_b = H(S,A\mid S') and ΔH=Hf−Hb\Delta H = H_f - H_b. Each of the 40 continuous variables (17 observation, 6 action, 17 next observation) is z-scored and placed into three equal-width bins whose edges are fixed from one reference agent's healthy window; statistics are computed on W=500W=500-step windows with stride 50. During a healthy period the mean and SD of each statistic are frozen; a statistic is "marked" when it leaves ±3σ\pm3\sigma, marks latch for the rest of the trial, and the monitor alarms once three of the four statistics are marked. The comparator is windowed reward under the same calibration, threshold and latch. On 21 frozen SAC/PPO HalfCheetah-v4 agents, eight step-change perturbation conditions (action noise, observation noise, external force, gravity) give 168 perturbation trials and 21 controls. The monitor detects 69.0% vs 44.0% for reward with 5/21 control alarms each; the whole-agent bootstrap advantage is 25.0 points [14.9,35.1][14.9, 35.1]. A within-cohort ablation at a matched five-control-alarm threshold compares normalisations (Table 3). Appendix E derives the elementary bound P≤12−I(S;A)/(2C)P \le \tfrac12 - I(S;A)/(2C). The authors state explicitly that no new formalism is introduced and that the operating point was selected in-sample.

Critical strengths
  1. Clear problem statement. Distinguishing "is the outcome still good" from "is the process producing it still the healthy one" is a sensible framing for runtime oversight of deployed agents, and the paper keeps the two apart consistently.
  2. Paired, matched design. Same windows, same calibration period, same latch and the same control-alarm count for every comparison; the Table 3 ablation at a matched false-alarm budget is the right way to compare candidate statistics. The leave-one-agent-out check is reported even though it is unfavourable.
  3. Unusually candid reporting. In-sample rule selection, exposure dependence of latched alarms, plug-in bias, representation dependence and seed screening are all stated plainly, with the numbers that quantify them.
  4. Cheap and reproducible. The monitor needs only the operational stream and a calibration period; logs, checkpoints and analysis scripts are released.
Critical weaknesses
  1. In the sampling regime used, the plug-in statistics measure symbol occupancy rather than "transition coupling", and the paper's own numbers show it. Windows contain W=500W=500 triplets, but the composite alphabets have 317≈1.3×1083^{17}\approx1.3\times10^8 observation symbols, 3233^{23} (S,A)(S,A) symbols and 3403^{40} triplet symbols. From the reported averages (C≈18.68C\approx18.68 bits, P≈0.334P\approx0.334, H(S)≈6.68H(S)\approx6.68 bits, H(A∣S)≈0.63H(A\mid S)\approx0.63 bits, endpoint term ≈0\approx0) one obtains I(S,A;S′)≈6.24I(S,A;S')\approx6.24 bits, H(S,A)≈7.3H(S,A)\approx7.3 bits, H(S,A,S′)≈7.75H(S,A,S')\approx7.75 bits, and hence Hf=H(S′∣S,A)≈0.44H_f = H(S'\mid S,A)\approx0.44 bits (this reproduces the reported ΔH≈−0.63\Delta H\approx-0.63). So a typical window has a perplexity of roughly 27.3≈1602^{7.3}\approx160 distinct (S,A)(S,A) cells for 500 samples, about three samples per cell, and the "forward uncertainty" of 0.44 bits is the in-sample conditional entropy of cells that mostly contain one to three observations. In that regime the plug-in conditional entropy is dominated by its negative bias (Paninski 2003); the Miller–Madow correction (m−1)/(2Nln⁡2)(m-1)/(2N\ln2) with mm occupied cells is of order 0.3–0.4 bits for the triplet entropy here, the same order as HfH_f itself. Correspondingly, I(S,A;S′)=H(S′)−HfI(S,A;S') = H(S') - H_f sits at 93%93\% of H(S′)H(S') simply because conditioning cells are nearly unique, which is what "P averages 87.1% of its ceiling" (Appendix E) is reporting. The consequence is that PP, HfH_f and HbH_b move whenever a perturbation changes how many distinct bin-symbols a window visits, independently of whether the dynamics have become more or less predictable. The paper's own diagnostic confirms this: under 3% observation noise H(S)H(S) rises from 6.68 to 7.63 bits, CC by 2.45 bits and II by 1.10 bits, so PP increases (Table 8: dP=+1.71dP=+1.71) and the noisiest sensor condition reads as more coupling; the authors attribute this to plug-in bias (Appendix D). The signs in Table 8 are mixed across conditions (dP>0dP>0 for 4% action noise and 3% observation noise, dP<0dP<0 elsewhere), so PP cannot be read as a share of the budget "expressed as coupling" that falls under decoupling. The finding that the better-sampled per-joint representation is less sensitive (Appendix B) points the same way: the sensitivity comes from sparse occupancy. The detections may be real, but the interpretive claim in the title and abstract (coupling, partial decoupling) is not supported by the estimator used. At minimum the paper needs (a) bias-corrected or coverage-adjusted estimators (Miller–Madow, Grassberger 1988, NSB, or a jackknife), (b) per-window occupancy counts and H(S,A,S′)H(S,A,S') relative to log⁡2W=8.97\log_2 W = 8.97, and (c) evidence that PP moves in the direction the interpretation predicts under each fault.

  2. The Table 3 ablation is largely a consequence of the saturation above, not evidence for the "budget". With I≈0.93 H(S′)I\approx0.93\,H(S'), the min-entropy normalisation I/min⁡{H(S,A),H(S′)}≈0.93I/\min\{H(S,A),H(S')\}\approx0.93 is pinned near one and has almost no room to move, so its 41.1% is a mechanical consequence of saturation rather than evidence against that normalisation in general. Conversely the budget denominator contains H(S)+H(A)H(S)+H(A), which do move under perturbation; dividing a nearly saturated numerator by a moving denominator produces an alarm whenever marginal entropies change. The "9 points over raw MI and 18 over min-entropy" headline (abstract, contributions) therefore does not isolate a property of the accounting. Note also that the plain conditional entropies HfH_f and HbH_b (60.7%) each outperform PP (58.9%) in the same table, so the "central construction" is not the best single reading.

  3. The only comparator is a scalar function of the monitor's own input, so "detects what reward misses" is close to tautological. In HalfCheetah-v4 the per-step reward is forward velocity minus 0.1∥a∥20.1\|a\|^2, and forward velocity is the root-xx velocity coordinate of the observation vector (up to the finite difference used in the reward). Windowed reward is thus a fixed one-dimensional smooth function of the (S′,A)(S',A) stream the monitor reads. A 40-variable change detector has strictly more information than one scalar projection of it, so higher detection at matched false alarms is expected for any reasonable multivariate drift detector. The paper acknowledges (Sections 2 and 7) that a two-sample test on the same stream is "the first comparison we would add", but without it the contribution of the specific statistics cannot be assessed. Cheap, reward-free, first-person baselines that belong in the paper: Hotelling T2T^2 or CUSUM on the z-scored 40-dimensional means; a window-vs-calibration χ2\chi^2 or KL distance on the same binned alphabet; MMD (Gretton et al. 2012); H(S)H(S) alone; and the one-step prediction error of a dynamics model (Haider et al. 2023, which is cited). Until one of these is shown to under-perform PP at matched false alarms, the empirical claim reduces to "the stream contains more information than the reward", which is not in doubt.

  4. The operating point is selected on the same 21 controls it is scored on, and the one validated procedure in the paper does not reproduce it. Leave-one-agent-out selection returns k=2k=2 in 20 of 21 folds and yields 80.4% detection at 11/21 control alarms (Appendix B): on the validated procedure the monitor's false-alarm rate is more than double reward's and the matched-false-alarm comparison no longer exists. The paper states this honestly but still headlines 69.0% vs 44.0% at 5/21 in the abstract and contributions. A validated number (nested selection of kk and zz on held-out agents, or a time-to-false-alarm on long nominal runs, which the authors say they do not have) is needed before the advantage can be quoted.

  5. The per-statistic thresholds in Table 3 are read off the gap between the fifth- and sixth-largest control excursions among 21, and the resulting differences carry no uncertainty. With agent-clustered intervals of ±10\pm10 points on the headline comparison, differences of 9 points (PP vs raw MI) and 2 points (PP vs I/H(S,A,S′)I/H(S,A,S')) are well within noise. Please report agent-bootstrap intervals for every row of Table 3 and the full detection–false-alarm curve rather than a single point per statistic.

  6. Latched marks over roughly 690 heavily overlapping windows make "5 of 21 controls" an exposure count rather than a rate, and the ±3σ\pm3\sigma rule has no nominal meaning. With W=500W=500 and stride 50, consecutive windows share 90% of their samples; σm\sigma_m is estimated from 291 such windows (about 30 effectively independent ones). The control alarms are built from single-window excursions (Appendix D: "1–16 of roughly 690 windows"). The paper acknowledges all of this, but it means the comparison is between two latched detectors at an arbitrary exposure. A sequential formulation with an average-run-length calibration (CUSUM, which is cited) is the standard tool and would give an exposure-free quantity.

  7. Cohort selection may favour the monitor's control-alarm count. Six of sixteen PPO seeds were excluded "on inspection of their unperturbed return traces" with no recorded criterion. Unstable nominal behaviour is exactly what inflates calibration variance and triggers control alarms, so excluding those seeds plausibly lowers the control-alarm count of both detectors, and not necessarily by the same amount. Reporting results on all 27 seeds (the authors say the runs are archived) would settle this.

  8. Single environment, one reference agent for bin edges, step-change faults of small and undefined magnitude. "1%, 3%, 4% Gaussian noise" is never defined (fraction of the action range? of the per-dimension SD?). Refitting bin edges per agent changes detection from 69.0% to 61.3%; the per-joint representation changes it again; a single agent's reference window fixes the edges for both algorithms. The reported rates are therefore properties of one representation choice rather than of the statistic. The confound between entry point and mechanism (stochastic agent-side versus systematic environment-side faults, Table 9) is acknowledged, and the three missing factorial cells are the obvious next experiment.

  9. No early-warning or localisation value, as the authors say. Median lead is zero; held-out-condition recall is 51.2% against 50% chance. These are fine as negative results, but they leave the use case beyond "a second change detector" unclear.

  10. Equation (2) and Appendix E are a two-line consequence of I≤min⁡{H(S,A),H(S′)}I\le\min\{H(S,A),H(S')\} and should not be presented as a result; the check that no window violates it only verifies code. A large share of the main text is disclaimers; the space would be better spent on the baselines of Weakness 3 and the estimator diagnostics of Weakness 1. The title's "task-agnostic" is redefined in Section 1 to mean "does not read reward", which is not what readers will take it to mean.

Questions for authors
  1. Please report, per window, the number of occupied cells for SS, (S,A)(S,A) and (S,A,S′)(S,A,S') and H(S,A,S′)H(S,A,S') relative to log⁡2W=8.97\log_2 W=8.97, and recompute PP, HfH_f, HbH_b with a bias-corrected estimator (Miller–Madow, Grassberger 1988, or NSB). Do the ordering in Table 3 and the 69.0% headline survive?
  2. In Table 8, dPdP is positive for 3% observation noise and 4% action noise and negative otherwise. Under the paper's interpretation, should a loosened agent–environment loop not lower the coupling share in every condition? If not, what is PP reading?
  3. Is windowed reward in HalfCheetah-v4 not a known scalar function of (S′,A)(S',A) (forward velocity minus control cost)? If so, how should "reward-independent" and the comparison in Section 5.2 be interpreted?
  4. What do a χ2\chi^2/KL distance between the window histogram and the calibration histogram on the same binned alphabet, an MMD on the raw 40-dimensional vectors, and H(S)H(S) alone detect at 5/21 control alarms?
  5. What are the results on all 27 trained seeds, and what is the detection of the leave-one-agent-out operating point at matched false alarms, if a matched point exists?
  6. Define the noise scale in Table 2. Are the SAC agents evaluated with the deterministic mean action, and if so, how does 1% Gaussian action noise compare with the policy's own training-time stochasticity?
  7. Could the authors run one long unperturbed trial per agent (e.g. 10610^6 steps) and report mean time to false alarm for both detectors?
3 comments
AuthorsOpenReview ↗

Response to Reviewer iTwq

Thank you for the careful review. We tested your central question directly: does 𝑃 respond to transition structure when visitation is held fixed?

Is 𝑃 reading the loop? (W1, Q1, Q2)

Healthy 500-step windows averaged 210 occupied 𝑆 cells, 268 (𝑆, 𝐴) cells and 343 triplets. Triplet coverage was 0.41, with 59% singletons and joint entropy 7.74 bits versus log2 𝑊 = 8.97. At comparable gridworld coverage, plug-in biases relative to exact values were about +2% for 𝑃 , −80% for 𝐻𝑓 and −62% for 𝐻𝑏. At fixed state–action visitation, a known dynamics change moved 𝑃 by −5.42 healthy-window SDs and 𝐻(𝑆) by −0.08. A visitation-only change moved them by +0.42 and −4.58 SDs, respectively.

In HalfCheetah, mean 𝑃 fell in all eight fault conditions with Chao–Shen estimates, and fell for 19–21 of 21 agents in the four strongest conditions. The positive changes in Q2 reversed under coverage adjustment.

What do the specific statistics add? (W2, W3, Q3, Q4)

For W2, min-entropy saturation and inconsistent handling of undefined windows affected Table 3. Correcting the script changes the min-entropy row from 41.1% to 50.6% and the joint-entropy row from 52.4% to 61.3%. With coverage adjustment, the tested normalizations detect 74–76% at the main operating point. We withdraw the 9- and 18-point headline. 𝐶 measures the three marginal uncertainties and gives the exact gap identity 𝑃 = (1/2) − [𝐻𝑓 + 𝐻𝑏 + 𝐼(𝑆; 𝐴)]/(2𝐶).

For W3/Q3, HalfCheetah reward uses true root-position change and the executed action. The logged stream contains neither root position nor the executed action under action noise.

For Q4 (App. D.2), at 5/21 controls in the descriptive comparison, joint histogram KL detected 72.0%, rawstream MMD 51.8%, and 𝐻(𝑆) 47.6% (three-of-four 70.2%, 𝑃 73.8%, reward 44.0%). At 6/21 held-out alarms, joint KL ties fixed three-of-four at 72.6% and KL on the state alone reaches 70.2%; MMD reaches 52.4% (61.3% at the best of the settings we tested), one-step prediction error 61.3% and 𝐻(𝑆) 50.0%. We also evaluated two-sided CUSUM on episode-wise reward and telemetry (App. D.6). Under known gridworld interventions, the normalized state and action shares 𝐼(𝑆; 𝑆′)/𝐶 and 𝐼(𝐴; 𝑆′ ∣ 𝑆)/𝐶 separated the tested randomising perturbations (action noise, slip) from the directionally biased ones (sensor offset, wind).

Operating point, exposure, seeds (W4–W7, Q5, Q7)

Thresholds were selected on the other agents, repeating selection in each bootstrap resample. At 6/21 held-out control alarms, fixed three-of-four detects 72.6%, 𝑃 74.4%, and reward 50.6%. We give intervals and curves (App. D.2–D.3) for every ablation row. In the 1,000-episode nominal reruns, mean time to first alarm restricted to 985 evaluation episodes was 495, 552 and 501 episodes, respectively. Each method alarmed on 6/21 runs by episode 100. Episode-wise CUSUM gave similar aggregate detection for telemetry and reward on different faults. Combining them detected 78.6–80.4% at 8/21 nominal alarms by episode 100, versus 6/21 for each standalone monitor. Across 28 logged runs, grouping the two evaluations of PPO seed 19, fixed three-of-four detected 65.2% and reward 43.8%, each at 7/28 control alarms.

Scope (W8–W10, Q6)

By “task-agnostic,” we mean that the statistic uses only state–action–next-state records, without access to rewards, goals, or a task model. We apply the same definition to discrete gridworld navigation and continuous HalfCheetah locomotion.

SAC and PPO used deterministic actions. Zero-mean Gaussian noise had SD 0.01/0.03/0.04 in action units (range [−1,1]) and 0.01/0.03 in normalized observation units. SAC’s stochastic action SD was 10–14 times the 0.01 action-noise level. Refitting bin edges per agent lowered detection from 69.0% to 61.3%. The nine-setting sweep shows how detection varies with bins and windows. Four gridworld cases cross agent/environment entry with randomising/directionally biased perturbations. A directionally biased actuator fault (gain loss) remains untested. Within one post-onset episode, 𝑃 caught 19.0 percentage points more faults than reward (95% interval [7.1,32.1]).

Localization remained at chance. Eq. (2) of the submitted version follows from the gap identity in Eq. (3) of the revision.

Thank you for the response and additional clarifications. My assessment of the paper is now positive, and I will raise my score.

AuthorsOpenReview ↗

Response to Reviewer iTwq

Thank you very much for your careful reading and for raising your score. Your review was the most detailed we received, and it shaped much of the revision: the coverage diagnostics and coverage-adjusted estimation, the same-stream baselines, held-out threshold selection, the long nominal runs and sequential comparison, and the corrections to the original ablation all trace back to your questions. The paper is clearer and better grounded because of them. Whatever the final decision, the work has benefited greatly from your review, and we are grateful for the time and rigour you put into it.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response to all reviewers and the area chair

We thank the reviewers for pressing us to test a distinction all three valued: task performance and the process producing it can change independently. The new results clarify where this distinction is useful and correct two entries in the original ablation.

  1. What the telemetry measures. App. B gives an information accounting for a declared discrete control record (𝑆, 𝐴, 𝑆′). The chain rule splits 𝐼(𝑆, 𝐴; 𝑆′) into 𝐼(𝑆; 𝑆′) and 𝐼(𝐴; 𝑆′ ∣ 𝑆). Dividing each by the marginal entropy budget 𝐶 > 0 gives the two shares of 𝑃 . The exact gap identity 𝑃 = 1 2 − [𝐻𝑓 + 𝐻𝑏 + 𝐼(𝑆; 𝐴)]/(2𝐶) yields the bound. The appendix also derives Δ𝐻 = 𝐻(𝑆′) − 𝐻(𝑆) − 𝐻(𝐴 ∣ 𝑆) and bounds the plug-in endpoint term. These observational summaries can stay constant under some changes and cannot identify a fault on their own.
  2. New evidence.
  • Gridworld (§5.1, App. C). At fixed state–action visitation, a known dynamics change moved 𝑃 by −5.42 healthy-window SDs and 𝐻(𝑆) by −0.08. A visitation-only change moved them by +0.42 and −4.58 SDs, respectively. The normalized shares separated the tested randomising perturbations from the directionally biased ones. At HalfCheetah-like coverage, plug-in biases were about +2% for 𝑃 , −80% for 𝐻𝑓 and −62% for 𝐻𝑏. We also tested gradual faults and changes in start region.
  • Held-out results (§5.2, App. D). With Chao–Shen estimates and thresholds selected on the other agents, fixed three-of-four detected 72.6%, 𝑃 alone 74.4%, and reward 50.6%, each at 6/21 held-out control alarms. Bootstrap intervals repeat threshold selection. Baselines include MMD, occupancy entropy, one-step prediction error, smoothed joint-histogram KL and CUSUM on episode reward.
  • Robustness. We evaluated all 28 logged runs and nine bin/window settings. Each retained agent was rerun for 1,000 nominal episodes. At the 25% calibration target for alarms within 100 episodes, each single monitor alarmed on 6/21 held-out runs. We report the fraction of runs alarmed by 50, 100, 200, 500 and 985 episodes (App. D.5–D.6).
  1. What the comparisons show. At 6/21 nominal alarms within 100 episodes for each standalone CUSUM monitor, three-of-four telemetry detected 68.5% and reward 70.8%. Telemetry caught 18 faults reward missed, and reward caught 22 telemetry missed. Telemetry-only catches included sensing and actuation disturbances, while reward-only catches included external forces. Joint KL also tied fixed three-of-four at 72.6% under the latched window protocol. The added value of the accounting is that its components show how the measured relationships shift, as the controlled gridworld cases illustrate. Combining sequential reward and telemetry detected 78.6–80.4%, with a higher nominal alarm count of 8/21 within 100 episodes.
  2. Corrections. In Table 3, undefined windows were counted as misses for two rows. Min-entropy detection changes from 41.1% to 50.6%, and joint-entropy detection from 52.4% to 61.3%. We withdraw the normalization advantage claim. Removing the reset triplet at each episode boundary changes held-out detection by at most two percentage points. We also correct the seed and fold descriptions. LLM feedback. The forum also shows the LLM feedback we requested at submission. It arrived while the submission window was still open, and we addressed it in the revision of 5 October, the version the reviewers assessed. Direction. The accounting uses only (𝑆, 𝐴, 𝑆′), so the same readings apply wherever an interaction loop is recorded; its decomposition is a first step toward attributing a change to the agent or to the environment without reward. That step needs observations or assumptions beyond the record, and we leave it to future work. Revision map: App. B (theory); §5.1 and App. C (gridworld); §3.2–3.5 (decomposition, estimation, held-out selection, baselines); §5.2–5.3 and App. D (held-out comparisons, long runs, sequential detection); App. D.7 (corrections). Code: https://anonymous.4open.science/r/idt_HalfCheetah-10F5/. 1
AuthorsOpenReview ↗

General response to all reviewers and the area chair

We thank the reviewers for pressing us to test a distinction all three valued: task performance and the process producing it can change independently. The new results clarify where this distinction is useful and correct two entries in the original ablation.

  1. What the telemetry measures.

App. B gives an information accounting for a declared discrete control record (𝑆, 𝐴, 𝑆′). The chain rule splits 𝐼(𝑆, 𝐴; 𝑆′) into 𝐼(𝑆; 𝑆′) and 𝐼(𝐴; 𝑆′ ∣ 𝑆). Dividing each by the marginal entropy budget 𝐶 > 0 gives the two shares of 𝑃 . The exact gap identity 𝑃 = (1/2) 2 − [𝐻𝑓 + 𝐻𝑏 + 𝐼(𝑆; 𝐴)]/(2𝐶) yields the bound. The appendix also derives Δ𝐻 = 𝐻(𝑆′) − 𝐻(𝑆) − 𝐻(𝐴 ∣ 𝑆) and bounds the plug-in endpoint term. These observational summaries can stay constant under some changes and cannot identify a fault on their own.

  1. New evidence.
  • Gridworld (§5.1, App. C). At fixed state–action visitation, a known dynamics change moved 𝑃 by −5.42 healthy-window SDs and 𝐻(𝑆) by −0.08. A visitation-only change moved them by +0.42 and −4.58 SDs, respectively. The normalized shares separated the tested randomising perturbations from the directionally biased ones. At HalfCheetah-like coverage, plug-in biases were about +2% for 𝑃 , −80% for 𝐻𝑓 and −62% for 𝐻𝑏. We also tested gradual faults and changes in start region.

  • Held-out results (§5.2, App. D). With Chao–Shen estimates and thresholds selected on the other agents, fixed three-of-four detected 72.6%, 𝑃 alone 74.4%, and reward 50.6%, each at 6/21 held-out control alarms. Bootstrap intervals repeat threshold selection. Baselines include MMD, occupancy entropy, one-step prediction error, smoothed joint-histogram KL and CUSUM on episode reward.

  • Robustness. We evaluated all 28 logged runs and nine bin/window settings. Each retained agent was rerun for 1,000 nominal episodes. At the 25% calibration target for alarms within 100 episodes, each single monitor alarmed on 6/21 held-out runs. We report the fraction of runs alarmed by 50, 100, 200, 500 and 985 episodes (App. D.5–D.6).

  1. What the comparisons show. At 6/21 nominal alarms within 100 episodes for each standalone CUSUM monitor, three-of-four telemetry detected 68.5% and reward 70.8%. Telemetry caught 18 faults reward missed, and reward caught 22 telemetry missed. Telemetry-only catches included sensing and actuation disturbances, while reward-only catches included external forces. Joint KL also tied fixed three-of-four at 72.6% under the latched window protocol. The added value of the accounting is that its components show how the measured relationships shift, as the controlled gridworld cases illustrate. Combining sequential reward and telemetry detected 78.6–80.4%, with a higher nominal alarm count of 8/21 within 100 episodes.

  2. Corrections. In Table 3, undefined windows were counted as misses for two rows. Min-entropy detection changes from 41.1% to 50.6%, and joint-entropy detection from 52.4% to 61.3%. We withdraw the normalization advantage claim. Removing the reset triplet at each episode boundary changes held-out detection by at most two percentage points. We also correct the seed and fold descriptions.

LLM feedback.

The forum also shows the LLM feedback we requested at submission. It arrived while the submission window was still open, and we addressed it in the revision of 5 October, the version the reviewers assessed.

Direction. The accounting uses only (𝑆, 𝐴, 𝑆′), so the same readings apply wherever an interaction loop is recorded; its decomposition is a first step toward attributing a change to the agent or to the environment without reward. That step needs observations or assumptions beyond the record, and we leave it to future work. Revision map: App. B (theory); §5.1 and App. C (gridworld); §3.2–3.5 (decomposition, estimation, held-out selection, baselines); §5.2–5.3 and App. D (held-out comparisons, long runs, sequential detection); App. D.7 (corrections). Code: https://anonymous.4open.science/r/idt_HalfCheetah-10F5/.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.