acceptodds
Withdrawn from ICLR 2027

Who Detects What? Budgeted Detection of Conditional Mechanism Shifts in Frozen-Policy RL Streams

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Reinforcement learning policies are increasingly deployed as frozen components, yet their environments can change beneath a stable surface: observation marginals remain approximately unchanged while a conditional reward or transition mechanism shifts. We study deployment-time detection: given a frozen-policy stream with a pre-registered episode-level false-alarm budget (alpha=0.05), which detector signal families actually detect which mechanism changes, and at what minimum injected intensity? In two MiniGrid environments we construct controlled mechanism injections—transition slip (seven intensities, a small-marginal-shift variant of the premise) and reward flip (four probabilities, the stable-marginal condition exactly)—verified by independent probes, and compare five detector families in a frozen 165-run matrix. Three findings emerge: (i) detection is mechanism-specific—marginal- and trajectory-statistics detectors are blind to reward flips at all intensities (entailed by their score construction, confirmed empirically), while the conditional discriminator, joint-most-sensitive on slip, is near-blind to weak flips that the value residual detects perfectly; (ii) an episode-stream circular-shift calibration is required for honest false-alarm control, as window-score-sequence shifting is provably weak; (iii) batch-style AUROC rankings do not substitute for budgeted deployment-level detection rates (rho=0.885; one persisted-score seat). A pre-registered hold-out on two unseen intensity levels passes all frozen criteria, replicating the specificity structure, as does a pre-registered cross-cell confirmation of the flip mechanism on the second environment; we release all streams, manifests, per-window artifacts, and the pre-registered protocol.

Primary area:
reinforcement learning
Keywords:
anomaly detection, reinforcement learning, drift detection, safe deployment, false alarm control, frozen policy, mechanism shift, MiniGrid

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_6C34OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies the detection of environmental changes after deploying a frozen reinforcement learning policy, with a focus on cases where the observation distribution changes little but the reward or state transition mechanism changes, i.e., the so-called “marginal-stable, conditional-mechanism shift,” and builds an evaluation framework for this setting. The contributions listed by the paper itself feel somewhat deliberate and limited. The paper does not attempt to apply its findings to a broader domain, but instead almost restricts itself to “we conducted some experiments and found these experimental results.”

Critical strengths

None.

Critical weaknesses

My main concern is that the paper provides only limited new insight. The clearest result of the paper can be summarized as follows: detectors that do not use reward information cannot detect changes that occur only in the reward. For detectors that already use reward information but still fail, the paper does not sufficiently distinguish whether this is a limitation of the detection principle itself or is caused by other unidentified reasons. I would rather summarize the paper as conducting some experiments and then analyzing those experiments themselves. I did not find sufficiently convincing insights that could guide method selection or improvement. For a paper whose main contribution is evaluation and analysis, I do not think the current depth of analysis is sufficient to support its claimed contribution.

The presentation does not quite follow common research conventions. There are many self-created expressions presented in a highly specialized or terminology form, which greatly increases the difficulty of reading, e.g., detector seat, honesty ledger, edge-stability, episode-level false-alarm budget, budgeted detection, and so on. BTW, the font also seems somewhat inconsistent with the default template and looks a bit weird. Similarly, for a paper whose main contribution is evaluation and analysis, I do not think the current clarity of presentation is easy for readers to understand its contribution, and apply it to their own work.

Questions for authors

I appreciate that the authors provided supplementary materials, but there seems to be little in them that actually helps with reproduction. Apart from the paper's TeX project, there is only one experimental results table, which I believe is incomplete, and the relatively important ep_far entries are all empty. For B4, when (p=0.3,0.5), the detection rates for all seeds are 1, which is inconsistent with the description in the main text. For C1, when the reward-flipping probability is only 0.1, the per-episode detection rates are also all 1, but it is unclear how this quantity is calculated. The csv is hard to read, maybe I read something wrong. Could the authors explain these issues and provide the actual source code? Of course, if the authors are unwilling to provide the source code, explaining these issues alone would also be acceptable. This point may only affect my Soundness score. My overall rating is mainly based on the Critical Weaknesses.

2 comments
AuthorsOpenReview ↗

we correct the B4 sentence, measure episode-level FAR after submission, identify the Empty-8x8 C1 cell as a threshold collapse, and provide the submitted code and data. Terminology, the code map and the two paper-level corrections are addressed below.

Terminology. The manuscript used several coined terms that added reading cost without adding precision. The revision replaces them: detector seat becomes detector or signal family; honesty ledger becomes registered limitations and negative-results log; edge stability or marginal stability becomes covariate-marginal stability; episode-level false-alarm budget becomes per-episode false-alarm budget; budgeted detection becomes detection under a fixed false-alarm budget.

Point 0, insight for method selection. We claim two checkable prescriptions, not a strong method. First, detector-mechanism matching: no single family covers both mechanisms; the pre-registered union of B5 and C1 recovers detection on both, with FAR reported only as a Bonferroni upper bound of 0.185 on slip, over budget, so it is not used for ranking. Second, calibration is not interchangeable: episode-stream circular shift is necessary while window-score-sequence shifting is provably a no-op, Propositions 1 and 2. Both are reusable, not proven optimal. The five-row matrix also contains only two rows with empirical information; the other three are constructive identities.

Point 1, B4 at p in {0.3, 0.5}. The reviewer is right and the text is wrong. Section 5.1 on page 7 says "B4 never reaches perfect detection below p = 0.7". In the submitted CSV, B4 is 1.000 at p = 0.3 and p = 0.5 for all three seeds. Table 1 showed five of the seven levels; the two omitted levels are exactly the ones that falsify the sentence. This is not a unit difference. The revision will rewrite Section 5.1 as "B4 is perfect at p >= 0.3 but drops to 0.867 plus or minus 0.189 at p = 0.15", show all seven levels, and tag CSV rows by experiment ID and design stage so the Section 3 identification ladder is not confused with the main matrix.

Point 2, C1 at q = 0.1. The table value is a window-level rate: the fraction of 15 half-overlapping windows, three seeds by five test windows, that alarm. The per-episode rate is also 1.0, but that is not evidence of sensitivity. On Empty-8x8 the calibration null has zero variance, the threshold collapses to the null value, and any small shift alarms. The post-submission benign control in supplement/post_submission/c1_benign_control shows the consequence: on Empty-8x8 the identity arm alarms at 0.000 while the reward-jitter arms average 0.518; on DoorKey, where the null is non-degenerate, identity is 0.030 and the strongest jitter arm reaches 0.140, mean over three checkpoints. The Empty C1 cell is a degenerate detector, not a strong one. The revision will mark it as a threshold collapse and state the window definition in the Table 1 caption.

Point 3, ep_far. The submitted CSV has ep_far empty in all 165 rows. B1 and B5 have no episode-level alarm rule by construction; B2, B4 and C1 have episode-level detection but the episode-level false-alarm rate was not persisted. That is a promise not kept, not a case of structural non-applicability. The post-submission analysis in supplement/post_submission/episode_far scores the 300-episode no-drift validation stream at the episode level: 14 of 15 measured cells are at or below 0.05; DoorKey C1 seed 1 reads 0.0533 with a Wilson 95 percent upper bound of 0.0849. C1 DoorKey seed 2 is not imputed because the on-disk checkpoint hash 8916ee does not match the run manifest ee286c; Empty-8x8 B2 and B4 have zero-variance nulls and are reported as degenerate. This is a non-rejection of alpha, not a tight certification.

Point 4, code and CSV. The supplement provides the submitted code used by the paper's experiments, per-run result.json files, the discriminator-family per-window scores, the 165-row per-seed table, the episode streams, and the three post-submission analyses. The per-seed table has a column map; the empty ep_far column is documented, and the post-submission episode-level table is kept separately under post_submission/episode_far, not merged into the submitted CSV.

Point 5, typesetting. Accepted. The re-laid PDF introduced a font that does not match the template; the revision will recompile with the template font.

Point 6, contribution type, items 7 to 10 and 18. We accept the characterization that the paper is primarily an evaluation and analysis. The revision states that only two of the five matrix rows carry empirical information; for each negative result we give two separable hypotheses, self-inclusion bias and ill-conditioned quantiles, and one discriminating experiment design. The parts that require new sampling are marked as not done, not written as conclusions.

Point 7, soundness. If the corrections and the supplement are accepted, we ask the reviewer to update the soundness judgment accordingly. The breadth of the setting remains a legitimate reason to lower the overall score, and we do not argue against it.

I do not believe this paper contributes any new knowledge or solves any problem. I will maintain my score, and any further attempts by the authors (or their agents) may be futile.

Reviewer_nJb4OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper monitors a frozen policy after deployment. The policy keeps acting and does not itself report a change. A separate monitor scores the recorded stream and raises an alarm when a window average exceeds a threshold set for a 5% false-alarm rate. The tested changes are a transition slip and a terminal-reward flip on MiniGrid, scored with five existing signals. When the reward flips and the observations stay the same, signals that do not read the reward never alarm, while the gap between predicted return and actual return alarms at every intensity. Reordering the finished window scores leaves that threshold unchanged. The threshold moves only when the episode stream is rotated and the windows are recomputed.

Critical strengths

The contribution is two results. When the observations remain unchanged, a reward change is detected only by the gap between predicted and actual return. Reordering window scores that have already been computed does not calibrate the alarm threshold; the episode stream must be rotated and the windows recomputed.

Critical weaknesses

The experimental scope is too small, and it remains unclear how this procedure should be applied to large-model reinforcement learning.

Questions for authors

Why do the authors consider this an important problem in the era of large models? I cannot see the importance. Could the authors give a concrete example showing that the problem they study matters?

2 comments
AuthorsOpenReview ↗

the paper argues for budget-based evaluation of deployment monitors. It does not claim LLM or VLA applicability; a token-level LLM stream remains future work, and the scope is limited to two MiniGrid environments.

Why this matters in the large-model era. The most damaging deployment-time failure of a post-training or RLHF agent is a reward-channel failure: the environment looks normal while the thing the agent's objective depends on is quietly broken. Our two mechanisms are minimal controlled versions of the two failure families reported in practice.

Reward flip, the marginal-stable axis. The task success of a deployed agent is scored by a checker or a reward model, and that score flips on some episodes: a site change, a sandbox policy change, or reward-model over-optimisation in a post-training evaluation loop. Input-distribution monitoring is silent. Table 1b shows B1, B2 and B4 at zero for all flip intensities, matching the pre-registered expected-failure hypothesis, while the value-residual channel alarms at the weakest tested level, q = 0.1. A monitor that looks only at observation statistics reports "all normal" while the objective is systematically corrupted.

Transition slip, the small-marginal-shift axis. The agent depends on a rarely used UI path or API route, and a backend change reroutes a small fraction of transitions. The marginals barely move; the conditional discriminator detects earliest, at epsilon-hat = 0.01875.

Operations need a budget, not a ranking. An on-call monitor has to respect a per-episode false-alarm budget, alpha = 0.05; otherwise it is muted within a week. That is why we report the smallest detectable intensity epsilon-hat under a fixed budget rather than an AUROC ordering, and why the gap between batch evaluation and deployment detection, rho = 0.885 in Section 5.4, is directly relevant to how detectors should be evaluated before deployment.

A frozen-policy, read-only monitor without drift labels is the deployment-side counterpart of learning-side concept-drift work. We do not claim that the paper applies to LLM or VLA deployment: the submitted paper lists LLM-agent streams as future work in Section 6 and does not claim to have evaluated one. A token-level LLM stream with an injected conditional shift is a separate study; we have no positive result of that kind and do not present one here. If the reviewer judges that this positioning does not fit the main conference, we accept evaluation as a measurement note rather than replacing readings with an importance narrative.

Scale. Two MiniGrid environments are a limitation, registered in Section 5.5(iv); the confirmations inside that boundary do not replace L3 or L4 evidence.

My main concern is that the experiments provided by the authors are too simplistic. Conclusions drawn under these experimental conditions are very difficult to apply to current LLM RL. I will maintain my score.

Reviewer_VGXTOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

To my understanding the paper proposes a change detection test in the context of a sequence of (state, action, reward) triples, potentially a two-sample test to identify whether the data has already changed when roll-out began. This is, for all intent and purposes, a special case of a regular sequence change detection problem and it would be much better to at least take advantage of the existing literature in this field.

This is also where my concerns regarding relevance start. There isn't really much explained about what the novelty in the paper is (there's literally no new method there), just an illustration of drift. The experiments are essentially three examples from GridWorld. It isn't clear whether even this limited algorithm would carry over cleanly.

Critical strengths

The paper is honest in its experimental description (including honest, pre-registered ledgers ... neatly described by AI).

Critical weaknesses

There's little novelty, no novel algorithm, weak experiments.

Questions for authors

Please make sure that your agent solves an important problem. This will lead to much better machine-generated papers.

1 comment
AuthorsOpenReview ↗

the paper is a controlled measurement study, not a new detector. The classical CUSUM and Page-Hinkley baseline was run after submission and is competitive rather than dominated; the scope remains limited to two environments and two mechanisms.

What the paper claims and does not claim. The paper does not propose a new detection algorithm; Section 1 says so directly: "we claim no new detector algorithm — the contribution is the protocol, the calibration principle, and the measured specificity structure." The object of study is the evaluation question itself: which detector families see which conditional mechanism change, at what injected intensity, under a pre-registered per-episode false-alarm budget. We do not treat protocol completeness as a contribution on its own; the protocol is there so that the readings can be checked. On the reward-flip axis the zero readings of B1 and B2 are constructive identities: their scores contain no reward information. The zero reading of B4 is marked empirical in the text, but it is not a positive finding either. Only B5 and C1 carry empirical information on that axis.

Relation to sequential change detection. Section 2 positions the work against multi-stream e-detectors, multi-stream quickest change detection, Model-X conditional change-point detection, conformal test martingales, and conditional drift tests. Before submission, CUSUM and Page-Hinkley were pre-registered as an expected-failure baseline and kept out of the main matrix. Following your comment we ran them on the same signal, the same calibration reference and the same test streams; the analysis is in supplement/post_submission/classical_baseline. At the weakest registered slip rung, p = 0.01875, CUSUM flags 0.344 of test episodes with realized validation FAR 0.047/0.097/0.107. B1 reaches 0.467 of windows, but its validation FAR in that cell is 0.20. On the same window geometry CUSUM reaches 0.667 plus or minus 0.340 against 0.467 plus or minus 0.189 for B1, and pays with validation FAR 0.600/0.000/0.200 against 0.200/0.200/0.200. The two trade detection for budget control in opposite directions; neither dominates. We therefore do not claim to beat the classical baseline.

The setting is not a trivial special case of sequential detection. A reward flip leaves the observation trajectory and its marginals structurally unchanged, so observation- and trajectory-based sequential tests are blind by construction. This was the pre-registered expected-failure hypothesis, confirmed in the matrix (B1, B2 and B4 zero at all four flip intensities, Table 1b), replicated at two unseen intensities (hold-out 30/30, Section 5.6) and in an untested mechanism-by-environment cell (Appendix D).

On the closing comment. The reference list retained notes from our source-verification workflow. The notes that actually appear in the submitted PDF are "v1 2025-10-21" on the eLife entry, "Version 3.1.0 used" on the MiniGrid entry, and "to appear" on one journal entry. They are not bibliographic data, and the revision removes them. Every reference was checked against its original source and remains verifiable. If there is a specific technical concern beyond the report, we are glad to answer it in the discussion.

Scale. Scale is the main limitation, registered in Section 5.5(iv): two environments, two mechanisms, five families and three seeds. The checks inside that boundary (hold-out, second-environment cross-cell confirmation, the third environment family MiniHack on the reward axis only) do not establish generalization to richer environments or to LLM-agent streams.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.