- Rating
- 2(+1)
- Confidence
- 1
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper addresses the stress test for controllers, done to increase the controller's failure rate by changing the time of a fixed disturbance. They conduct a comparison of stress tests and use the joint law of scheduling time and pre-intervention activity.
Critical strengths
- The paper is theoretically sound and provides bounds on the reward difference and TV norms. I also appreciate solving an optimization problem to find the w value, rather than relying on some heuristic.
Critical weaknesses
-
The paper is very hard to read, and the terms used are not natural. The flow is very difficult to follow, and the paper would require more motivational notes. For example, the contributions listed are very specific and may not be familiar to general ML readers. Moreover, the figure used does not ease the understanding ot eh apper. I would also suggest improving the font and equation editing for it. I believe the paper can do much better with a new write-up that accounts for the motivation and is more high-level.
-
I think FrozenLake and CartPole are very simple environments, and I would appreciate more ablations on larger, high-dimensional domains.
Questions for authors
- Are we dealing specifically with RL controllers? Can we work with model-based and other, more general controllers?
- How fast is it to solve the formulated linear program?
- How much time is needed to conduct this test, and how often does it need to be conducted?
2 comments
I reiterate my limited knowledge of this paper.
Response to hRWW: readability, larger environments, controller types and audit cost
Thank you for the review and for the concrete questions. We have rewritten the opening of the paper around the motivating problem, added a higher-dimensional continuous-control task, and separated the optimization time from the cost of collecting audit data.
W1. Readability.
We revised the presentation with a general ML reader in mind:
W2. Larger environments and ablations.
We added MuJoCo Hopper-v5 with ten frozen proximal policy optimization (PPO) controllers (two 64-unit hidden layers), one fixed horizontal-velocity impulse and a 250-step test horizon. All ten seeds are reported, including those that fall back to the reference. At the same audit cap of 45.5 million environment steps, the shared mixture gives a mean failure increase of 2.175 percentage points (95% seed-bootstrap interval 0.709 to 3.942), versus 1.079 points for the categorical mixture; six of ten controllers receive positive certificates (Section 5.6; Appendix G.1–G.2).
For ablations, Table 26 compares single-candidate versus mixture selection and generic versus survival-based reference bounds. Appendix G.4 adds a learned PPO stopping rule to the frozen library and reports its raw law discrepancy, corrected gains and additional cost. The finite-state tasks keep a distinct role: because their dynamics are known exactly, they let us verify the stated law and risk guarantees. All 960 original finite-state method/allocation rows satisfy those checks (Appendix D.2).
The finite-sample law bound and the certificate LP depend on the number of schedule cells and candidates, not on the state dimension. Candidate construction, score evaluation and simulation still depend on the task, so we do not claim dimension-free computational cost.
Q1. Is the method specific to RL controllers?
No. The audit treats the controller as a frozen black box that can be simulated, with any controller memory included in the simulated state. In the original experiments, 240 of the 250 controllers were obtained by dynamic programming on estimated transition models; only the ten CartPole controllers used model-free PPO. The Hopper controllers also use PPO. Model-based, rule-based and other frozen controllers fit the same interface whenever the simulator and independent-episode assumptions hold. Section 3 now states this explicitly.
Q2. How fast is the linear program?
The LP is small. With four non-reference candidates it has K+m variables: 36 for the 16-slot FrozenLake, CartPole and Hopper schedules, and 52 for the resource tasks. In the original controlled experiments, constructing the confidence bounds and solving the LP took 1.204 ms (FrozenLake) and 0.979 ms (CartPole) on average on an Apple M4 CPU. Because this timer also covers bound construction, we report it as such rather than as a pure LP benchmark (Appendix I, Table 33).
Q3. How long does an audit take, and how often is it needed?
For the original larger allocation, the recorded audit stage averages 0.895 s per FrozenLake configuration and 6.759 s per CartPole controller after kernel warm-up. This includes nominal-law processing, intervention rollouts and optimization; it excludes warm-up, controller training, score fitting, prefix calibration and final tests. Table 7 gives the cost in simulated transitions, and Table 33 states the timing scope.
An audit is run once for a given controller, simulator, initial-state distribution, shock, schedule and frozen library, and must be redone or revalidated if any of these changes. Running the selected tester afterwards needs only its score evaluations and simulated episodes, not a new LP. These are offline certification costs; the audit is not an online monitor.
Thank you again for the feedback. We hope the revised version is substantially easier to follow.