- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies the detection of environmental changes after deploying a frozen reinforcement learning policy, with a focus on cases where the observation distribution changes little but the reward or state transition mechanism changes, i.e., the so-called “marginal-stable, conditional-mechanism shift,” and builds an evaluation framework for this setting. The contributions listed by the paper itself feel somewhat deliberate and limited. The paper does not attempt to apply its findings to a broader domain, but instead almost restricts itself to “we conducted some experiments and found these experimental results.”
Critical strengths
None.
Critical weaknesses
My main concern is that the paper provides only limited new insight. The clearest result of the paper can be summarized as follows: detectors that do not use reward information cannot detect changes that occur only in the reward. For detectors that already use reward information but still fail, the paper does not sufficiently distinguish whether this is a limitation of the detection principle itself or is caused by other unidentified reasons. I would rather summarize the paper as conducting some experiments and then analyzing those experiments themselves. I did not find sufficiently convincing insights that could guide method selection or improvement. For a paper whose main contribution is evaluation and analysis, I do not think the current depth of analysis is sufficient to support its claimed contribution.
The presentation does not quite follow common research conventions. There are many self-created expressions presented in a highly specialized or terminology form, which greatly increases the difficulty of reading, e.g., detector seat, honesty ledger, edge-stability, episode-level false-alarm budget, budgeted detection, and so on. BTW, the font also seems somewhat inconsistent with the default template and looks a bit weird. Similarly, for a paper whose main contribution is evaluation and analysis, I do not think the current clarity of presentation is easy for readers to understand its contribution, and apply it to their own work.
Questions for authors
I appreciate that the authors provided supplementary materials, but there seems to be little in them that actually helps with reproduction. Apart from the paper's TeX project, there is only one experimental results table, which I believe is incomplete, and the relatively important ep_far entries are all empty. For B4, when (p=0.3,0.5), the detection rates for all seeds are 1, which is inconsistent with the description in the main text. For C1, when the reward-flipping probability is only 0.1, the per-episode detection rates are also all 1, but it is unclear how this quantity is calculated. The csv is hard to read, maybe I read something wrong. Could the authors explain these issues and provide the actual source code? Of course, if the authors are unwilling to provide the source code, explaining these issues alone would also be acceptable. This point may only affect my Soundness score. My overall rating is mainly based on the Critical Weaknesses.
2 comments
I do not believe this paper contributes any new knowledge or solves any problem. I will maintain my score, and any further attempts by the authors (or their agents) may be futile.
we correct the B4 sentence, measure episode-level FAR after submission, identify the Empty-8x8 C1 cell as a threshold collapse, and provide the submitted code and data. Terminology, the code map and the two paper-level corrections are addressed below.
Terminology. The manuscript used several coined terms that added reading cost without adding precision. The revision replaces them: detector seat becomes detector or signal family; honesty ledger becomes registered limitations and negative-results log; edge stability or marginal stability becomes covariate-marginal stability; episode-level false-alarm budget becomes per-episode false-alarm budget; budgeted detection becomes detection under a fixed false-alarm budget.
Point 0, insight for method selection. We claim two checkable prescriptions, not a strong method. First, detector-mechanism matching: no single family covers both mechanisms; the pre-registered union of B5 and C1 recovers detection on both, with FAR reported only as a Bonferroni upper bound of 0.185 on slip, over budget, so it is not used for ranking. Second, calibration is not interchangeable: episode-stream circular shift is necessary while window-score-sequence shifting is provably a no-op, Propositions 1 and 2. Both are reusable, not proven optimal. The five-row matrix also contains only two rows with empirical information; the other three are constructive identities.
Point 1, B4 at p in {0.3, 0.5}. The reviewer is right and the text is wrong. Section 5.1 on page 7 says "B4 never reaches perfect detection below p = 0.7". In the submitted CSV, B4 is 1.000 at p = 0.3 and p = 0.5 for all three seeds. Table 1 showed five of the seven levels; the two omitted levels are exactly the ones that falsify the sentence. This is not a unit difference. The revision will rewrite Section 5.1 as "B4 is perfect at p >= 0.3 but drops to 0.867 plus or minus 0.189 at p = 0.15", show all seven levels, and tag CSV rows by experiment ID and design stage so the Section 3 identification ladder is not confused with the main matrix.
Point 2, C1 at q = 0.1. The table value is a window-level rate: the fraction of 15 half-overlapping windows, three seeds by five test windows, that alarm. The per-episode rate is also 1.0, but that is not evidence of sensitivity. On Empty-8x8 the calibration null has zero variance, the threshold collapses to the null value, and any small shift alarms. The post-submission benign control in supplement/post_submission/c1_benign_control shows the consequence: on Empty-8x8 the identity arm alarms at 0.000 while the reward-jitter arms average 0.518; on DoorKey, where the null is non-degenerate, identity is 0.030 and the strongest jitter arm reaches 0.140, mean over three checkpoints. The Empty C1 cell is a degenerate detector, not a strong one. The revision will mark it as a threshold collapse and state the window definition in the Table 1 caption.
Point 3, ep_far. The submitted CSV has ep_far empty in all 165 rows. B1 and B5 have no episode-level alarm rule by construction; B2, B4 and C1 have episode-level detection but the episode-level false-alarm rate was not persisted. That is a promise not kept, not a case of structural non-applicability. The post-submission analysis in supplement/post_submission/episode_far scores the 300-episode no-drift validation stream at the episode level: 14 of 15 measured cells are at or below 0.05; DoorKey C1 seed 1 reads 0.0533 with a Wilson 95 percent upper bound of 0.0849. C1 DoorKey seed 2 is not imputed because the on-disk checkpoint hash 8916ee does not match the run manifest ee286c; Empty-8x8 B2 and B4 have zero-variance nulls and are reported as degenerate. This is a non-rejection of alpha, not a tight certification.
Point 4, code and CSV. The supplement provides the submitted code used by the paper's experiments, per-run result.json files, the discriminator-family per-window scores, the 165-row per-seed table, the episode streams, and the three post-submission analyses. The per-seed table has a column map; the empty ep_far column is documented, and the post-submission episode-level table is kept separately under post_submission/episode_far, not merged into the submitted CSV.
Point 5, typesetting. Accepted. The re-laid PDF introduced a font that does not match the template; the revision will recompile with the template font.
Point 6, contribution type, items 7 to 10 and 18. We accept the characterization that the paper is primarily an evaluation and analysis. The revision states that only two of the five matrix rows carry empirical information; for each negative result we give two separable hypotheses, self-inclusion bias and ill-conditioned quantiles, and one discriminating experiment design. The parts that require new sampling are marked as not done, not written as conclusions.
Point 7, soundness. If the corrections and the supplement are accepted, we ask the reviewer to update the soundness judgment accordingly. The breadth of the setting remains a legitimate reason to lower the overall score, and we do not argue against it.