acceptodds
Under review as a conference paper at ICLR 2027

SameShock: Auditing Adaptive Stress Tests at Matched Intervention Exposure

Abstract

An adaptive stress test can increase a controller's failure rate by changing when a fixed disturbance occurs. A valid comparison must distinguish harmful state-dependent timing from changes in the distribution of intervention times and in how often an intervention reaches an active system. We formulate this comparison through the joint law of scheduled time and pre-intervention activity, and develop a finite-sample audit for a frozen library of causal stressors. Untouched nominal trajectories estimate every candidate's intervention law by integrating its stopping probabilities; separate intervention rollouts estimate failure risk. A small linear program selects an episode-level mixture that certifies both a prescribed total-variation tolerance and a lower bound on excess failure probability after mixture selection. On 80 FrozenLake controllers, matching time alone permits an exact risk-maximizing tester to increase active exposure by 8.95 percentage points. Controlled audits separate savings from path sharing, modest gains from stopping integration, and gains from a sharper reference bound. On ten proximal policy optimization controllers in a continuous-state CartPole test, SameShock exposes a mean 2.40 percentage-point failure increase at a 1% joint-law tolerance, using a strict matched transition budget. At a lower impulse strength, independent-shock failure falls to 57.15% and the audited failure increment remains 2.28 percentage points.

Primary area:
reinforcement learning
Keywords:
adaptive stress testing, reinforcement learning, finite-sample certification, optimal stopping, simulation-based evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_hRWWOpenReview ↗
Rating
2(+1)
Confidence
1
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper addresses the stress test for controllers, done to increase the controller's failure rate by changing the time of a fixed disturbance. They conduct a comparison of stress tests and use the joint law of scheduling time and pre-intervention activity.

Critical strengths
  • The paper is theoretically sound and provides bounds on the reward difference and TV norms. I also appreciate solving an optimization problem to find the w value, rather than relying on some heuristic.
Critical weaknesses
  • The paper is very hard to read, and the terms used are not natural. The flow is very difficult to follow, and the paper would require more motivational notes. For example, the contributions listed are very specific and may not be familiar to general ML readers. Moreover, the figure used does not ease the understanding ot eh apper. I would also suggest improving the font and equation editing for it. I believe the paper can do much better with a new write-up that accounts for the motivation and is more high-level.

  • I think FrozenLake and CartPole are very simple environments, and I would appreciate more ablations on larger, high-dimensional domains.

Questions for authors
  • Are we dealing specifically with RL controllers? Can we work with model-based and other, more general controllers?
  • How fast is it to solve the formulated linear program?
  • How much time is needed to conduct this test, and how often does it need to be conducted?
2 comments
AuthorsOpenReview ↗

Response to hRWW: readability, larger environments, controller types and audit cost

Thank you for the review and for the concrete questions. We have rewritten the opening of the paper around the motivating problem, added a higher-dimensional continuous-control task, and separated the optimization time from the cost of collecting audit data.

W1. Readability.

We revised the presentation with a general ML reader in mind:

  • The abstract and introduction now start from a simple scenario, one fixed push applied to a controller, and explain why a push scheduled after an episode has ended changes effective exposure.
  • Section 3 now contains the two-type example (previously only in Appendix A.1), which shows in a few lines why matching time alone is insufficient.
  • The contributions are condensed into three short bullets, Figure 1 separates the independent audit from selection and deployment, and Appendix I adds a notation table.
  • The theorem statements and proofs are unchanged.

W2. Larger environments and ablations.

We added MuJoCo Hopper-v5 with ten frozen proximal policy optimization (PPO) controllers (two 64-unit hidden layers), one fixed horizontal-velocity impulse and a 250-step test horizon. All ten seeds are reported, including those that fall back to the reference. At the same audit cap of 45.5 million environment steps, the shared mixture gives a mean failure increase of 2.175 percentage points (95% seed-bootstrap interval 0.709 to 3.942), versus 1.079 points for the categorical mixture; six of ten controllers receive positive certificates (Section 5.6; Appendix G.1–G.2).

For ablations, Table 26 compares single-candidate versus mixture selection and generic versus survival-based reference bounds. Appendix G.4 adds a learned PPO stopping rule to the frozen library and reports its raw law discrepancy, corrected gains and additional cost. The finite-state tasks keep a distinct role: because their dynamics are known exactly, they let us verify the stated law and risk guarantees. All 960 original finite-state method/allocation rows satisfy those checks (Appendix D.2).

The finite-sample law bound and the certificate LP depend on the number of schedule cells and candidates, not on the state dimension. Candidate construction, score evaluation and simulation still depend on the task, so we do not claim dimension-free computational cost.

Q1. Is the method specific to RL controllers?

No. The audit treats the controller as a frozen black box that can be simulated, with any controller memory included in the simulated state. In the original experiments, 240 of the 250 controllers were obtained by dynamic programming on estimated transition models; only the ten CartPole controllers used model-free PPO. The Hopper controllers also use PPO. Model-based, rule-based and other frozen controllers fit the same interface whenever the simulator and independent-episode assumptions hold. Section 3 now states this explicitly.

Q2. How fast is the linear program?

The LP is small. With four non-reference candidates it has K+m variables: 36 for the 16-slot FrozenLake, CartPole and Hopper schedules, and 52 for the resource tasks. In the original controlled experiments, constructing the confidence bounds and solving the LP took 1.204 ms (FrozenLake) and 0.979 ms (CartPole) on average on an Apple M4 CPU. Because this timer also covers bound construction, we report it as such rather than as a pure LP benchmark (Appendix I, Table 33).

Q3. How long does an audit take, and how often is it needed?

For the original larger allocation, the recorded audit stage averages 0.895 s per FrozenLake configuration and 6.759 s per CartPole controller after kernel warm-up. This includes nominal-law processing, intervention rollouts and optimization; it excludes warm-up, controller training, score fitting, prefix calibration and final tests. Table 7 gives the cost in simulated transitions, and Table 33 states the timing scope.

An audit is run once for a given controller, simulator, initial-state distribution, shock, schedule and frozen library, and must be redone or revalidated if any of these changes. Running the selected tester afterwards needs only its score evaluations and simulated episodes, not a new LP. These are offline certification costs; the audit is not an online monitor.

Thank you again for the feedback. We hope the revised version is substantially easier to follow.

I reiterate my limited knowledge of this paper.

Reviewer_dTFYOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

I honestly couldn't figure out what this paper is supposed to be about.

Critical strengths

None

Critical weaknesses

As we all know, the conference has a high volume of sloppy AI assisted papers. This is a particularly bad offender and I didn't spend much time considering its merits, as it is clearly well below publication standard.

Questions for authors

None

1 comment
AuthorsOpenReview ↗

Response to dTFY: research question, revised presentation and new evidence

Thank you for your time. We have revised the presentation so that the research question is stated up front: the abstract and introduction are rewritten, the simplest example now appears in Section 3, Figure 1 is simplified, and a notation table is added.

The question. The paper studies how to compare stress testers that choose when to apply one fixed disturbance to a fixed controller. A tester can find more failures for two reasons: it may pick more vulnerable states, or it may simply deliver more disturbances while the system is still running (a disturbance scheduled after an episode has ended has no effect). Matching only the distribution of scheduled times leaves the second effect uncontrolled. We therefore compare testers through the joint distribution of the scheduled time and whether the system is active just before it, keeping every initial episode in the denominator.

The method. SameShock uses shared undisturbed trajectories to audit a frozen library of candidate testers, and separate shocked rollouts to estimate their failure risks. A small linear program selects an executable mixture of testers, with simultaneous finite-sample bounds on both its exposure discrepancy and its risk gain (Theorem 2; proofs in Appendix A).

New evidence. The revision adds ten-controller MuJoCo Hopper experiments and a covered full-path importance-sampling comparison. In Hopper, the shared mixture gives a mean failure increase of 2.175 percentage points (95% seed-bootstrap interval 0.709 to 3.942; Section 5.6, Appendix G). Appendix H separates reference-risk estimation from matched executable testing.

The use of AI assistance is disclosed in the manuscript's AI usage statement. All reported numbers come from executed simulations and exact finite-state calculations. If any specific claim, proof step or experiment appears unsound to you, we would be grateful if you could point it out so that we can address it directly.

Reviewer_FsodOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper considers adaptive stress testers that use a causal stopping rule to choose when to apply one fixed physical shock to a frozen controller. It argues that fair comparison with independent random timing requires matching the joint law of the scheduled time and whether the system is still active before the shock. Matching the time law alone lets a tester change how often shocks reach an active system. The authors introduce SameShock, which freezes a library of candidate hazards, estimates every candidate's joint time/activity law and failure risks, and solves an LP that select a mixture that certify lower bound on excess failure probability. The proposed approach is validated in case studies.

Critical strengths
  1. The paper studies adaptive stress tests and shows that matching the joint time/activity rules out the exposure changes that time matching permits
  2. It provides mathematical analysis and experimental validation.
Critical weaknesses
  1. The paper may benefit from discussing alternative approaches based on importance sampling. For example, Feng et al. (Nat. Commun. 2021) use adversarial adjustments to choose when to act while modifying estimates via importance sampling.

  2. The evaluation is small in scale, and its practical significance is unclear. The setting is a single fixed shock, two stylized MDPs, FrozenLake and CartPole. Audited gains are about 0.7–2.8 pp, and for CartPole (+2) only 5/10 controllers certify a gain above the tolerance.

Questions for authors
  1. What downstream decision does the certified gain G inform? Can you show a setting where SameShock changes a ranking of controllers or testers relative to time-only matching or independent timing?
  2. When is matching the intervention law preferable to importance-sampling/likelihood-ratio approaches that reweight to the nominal law (e.g., Feng et al. 2021)?
3 comments
AuthorsOpenReview ↗

Response to Fsod (1/2): matching versus importance sampling

Thank you for the careful and constructive review, and for pointing out the missing comparison with importance sampling. We agree that it deserved explicit treatment. The revision now distinguishes three quantities (Section 2, Appendix H) and adds two comparisons.

W1 / Q2. When is matching preferable to importance sampling?

The two approaches answer different questions. In Feng et al. (2021), the adversarial environment serves as a proposal for estimating failure under the naturalistic reference law; with coverage, full-path likelihood ratios recover that reference risk. In our single-shock setting, reweighting back to independent timing therefore estimates the independent-timing failure probability R0.

SameShock instead seeks an executable, state-aware tester whose own failure probability exceeds R0, while the joint law of scheduled time and pre-shock activity stays within a stated total-variation (TV) tolerance of the reference. The audit certifies both this constraint and a lower bound on the tester's risk increase. Importance sampling is the natural tool when the goal is an unbiased estimate of reference risk; matching is the natural tool when the goal is a deployable tester whose additional failures are measured at comparable exposure. We view the two as complementary.

New experiment: covered full-path importance sampling (Appendix H.3, Table 31).

We ran 20,000 fresh episodes per estimator on each of the 80 original FrozenLake controller configurations (3.2 million new episodes in total). The proposal mixes 10% independent timing with 90% unconstrained stopping, which preserves coverage and bounds the likelihood ratios by 10. Crude Monte Carlo and ordinary full-path importance sampling both target R0:

  • Exact mean reference failure: 3.2108%
  • Crude Monte Carlo: 3.2304% (conditional Monte Carlo standard error 0.0137 pp)
  • Full-path importance sampling: 3.2022% (standard error 0.0373 pp)

Importance sampling recovers the reference target even though the proposal's unweighted failure rate is 14.3527%. Its mean empirical effective sample size is 10.69% of the sample count, and for this proposal the exact estimator variance is 7.390 times that of crude Monte Carlo at equal episode counts. This proposal was designed for failure discovery rather than variance reduction, so the result is not an efficiency comparison with NADE.

Reweighting only the exposure cells (Appendix H.2, Table 30).

An intermediate option is to reweight only the scheduled-time/activity cells. This differs from full-path reweighting: it keeps the proposal's conditional choice of states within each cell. Across the 80 configurations, the mean standardized increase is 10.687 pp, compared with a mean exact joint-matched optimum of 5.775 pp. The standardized increase exceeds the corresponding exact optimum in every configuration. The cell-standardized quantity therefore cannot in general be read as the risk of an executable, exactly matched tester. This comparison uses the zero-tolerance optimum and is kept separate from the 1% tolerance of the original audit.

Summary. Full-path importance sampling estimates reference risk; cell standardization reports a conditional reweighted quantity; SameShock returns a causal tester together with an exposure certificate and a certified risk gain. Section 2 now states this distinction explicitly, and the new analyses keep candidate search separate from independent auditing, as in the original design.

Reference: Feng et al., Nature Communications 12, 748 (2021).

AuthorsOpenReview ↗

Response to Fsod (2/2): Hopper experiments and a certified risk-ranking change

W2. Scale and practical significance.

We agree that the original evaluation was small. We added standard MuJoCo Hopper-v5 experiments with ten frozen proximal policy optimization (PPO) controllers (two 64-unit hidden layers). The test applies one fixed horizontal-velocity impulse over a 250-step horizon, with four frozen score candidates plus independent timing; the impulse strength was selected on separate development controllers only. The audit algorithm, the 1% joint TV tolerance and the 5% per-controller audit error probability are unchanged (Section 5.6; Appendix G.1–G.2).

Results on all ten controllers, including those that fall back to the reference:

  • Shared mixture: +2.175 pp failure increase (95% paired seed-bootstrap interval 0.709 to 3.942); 6/10 positive certificates; mean per-controller certified lower bound 1.790 pp.
  • Categorical mixture: +1.079 pp (0.375 to 1.928).
  • Paired difference, shared minus categorical: 1.096 pp (0.322 to 2.013; Appendix G.2, Table 26).

The per-controller certificates and the across-controller bootstrap interval quantify different uncertainties. Both audits share a cap of 45.5 million environment steps; realized costs average 39.484 (categorical) and 40.295 (shared) million steps, so we do not claim a lower realized cost for the shared audit. Candidate construction and held-out tests are reported separately.

Appendix G.4 also adds a separately trained PPO stopping rule. Its raw failure increase is large, but it changes the intervention law substantially. After correction, the reference-plus-learned rule gives 0.490 pp, and the expanded six-component library gives 2.268 pp with eight positive certificates; Table 29 reports both intervals and the additional audit/replay cost. The original five-component experiment remains the primary comparison.

On the size of the gains: for FrozenLake, Table 3 separates controller capability from failure discovery. Penalty-8 controllers, for example, have 82.36% nominal completion and 1.88% independent-shock failure, and the audited tester adds 2.28 failure points, more than doubling the failure rate. This relative view complements, rather than replaces, the absolute effect. For CartPole +2, the 5/10 count uses the stricter criterion G > ε, under which a positive residual remains even after removing any cell-frequency contribution (Section 4); the shared mixture's mean certified lower bound there is 2.19 pp (Table 4).

Q1. What decision does G inform?

G lower-bounds the extra failure risk exposed by the selected executable tester at the stated exposure tolerance. It supports comparing frozen testers and checking whether risk rankings under independent timing persist under certified adaptive testing.

The revision gives a concrete example with simultaneous held-out uncertainty (Section 5.6; Appendix G.3, Table 28). Hopper controllers 917003 and 917005 have independent-timing failure rates of 19.36% and 22.74%. Under their deployed audited mixtures, the risks are 24.22% and 22.74%, reversing the order; the second controller falls back to the reference.

We examined all 45 controller pairs. Clopper–Pearson intervals with a total 5% error budget across all 50 component risks yield simultaneous intervals for the independently selected mixtures. Exactly one pair has disjoint intervals in opposite orders. We present this as a change in the risk comparison under the selected certified testers, not as a claim about globally optimal attacks.

We also retain the original utility-filter selection result in Section 5.4, where the selected deployment controller did not change. The new example addresses the risk-ranking part of your question without recasting that null result as a deployment benefit.

Thank you again; these two questions substantially sharpened the paper.

Thank you for providing new experiments and answers to questions. The responses helped clarify the different objectives in comparison with importance sampling-based work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.