- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper considers the problems of detecting violations of barrier conditions expressed via supermartingales at runtime. Their approach combines existing results from runtime monitoring and betting-based sequential testing to address the specific problem. The approach is evaluated on several benchmarks.
Critical strengths
-
The paper considers a problem that, to the best of my knowledge, has not been much explored in the stochastic setting
-
The approach provided by the authors is sound, and the experiments show the effectivness of the approach
Critical weaknesses
-
Much of the statistical machinery follows established betting-based sequential-testing methodologies, such as those used by Timans et al. (UAI 2025). While this is not a weakness in itself, it limits the methodological novelty. In addition, [Feng et al. "Runtime safety and reach-avoid prediction of stochastic systems via observation-aware barrier functions." AAAI 2026] also develop barrier-based runtime safety and reach-avoid prediction for stochastic systems. Although the objective differs from that in this paper, this relevant work should be discussed and the distinction clarified.
-
Parts of Section 3, particularly Section 3.2, present the framework very generally, without consistently connecting the abstract construction to the specific monitoring problem. Making these connections more explicit would improve the readability of that Section, which is currently not easy to read, especially for readers unfamiliar with sequential testing. Furthermore, the authors should clarify how the parameters required by the theory are selected across the benchmarks and strategies.
-
The main text reports ground truth for the control benchmarks but does not provide a corresponding assessment for the neural experiments.
-
There are some typos in the notation. For instance, in the Supermartingale definition, l is used both as the extreme of the interval and as a drift bound and x_T_n is sometimes called x_n
Questions for authors
-
Can you clarify the differences with [Feng et al. "Runtime safety and reach-avoid prediction of stochastic systems via observation-aware barrier functions." AAAI 2026] and the main technical contributions compared to state of the art?
-
Can you clarify how how the parameters required by the theory are selected across the benchmarks and strategies?
-
Can you report the ground truth also for neural networks and also clearly discuss/report the conservativism of the approach in the various experiments?
2 comments
We thank the reviewer again for the valuable feedback and comments.
We have now incorporated the changes described in our rebuttal into the revised paper, which we posted as an official comment. The changes are highlighted. Specifically:
-
[Q1] Related work: added the discussion of Feng et al. and the distinction from our monitoring problem in Section 5 (p. 10).
-
[Q2] Parameter selection: expanded Appendix E with the parameter choices, their rationale, and benchmark-specific score bounds (pp. 29–33).
-
[Q3] Neural ground truth: added the explanation in the main text (p. 8) and clarified the status of the monitored conditions under the adapted dynamics (p. 32).
-
[Q3] Conservatism: added discussion of drift slack and loss normalisation (p. 31), alongside the theoretical discussion (p. 27) and a CartPole comparison of certificate conditions (pp. 54–55).
Please let us know if any concerns remain or if we could clarify anything further. If our response and revisions address your concerns, we would be grateful if you would consider updating your score.
We thank the reviewer for the valuable feedback and comments.
[Q1] We clarify that [Feng et al., 2026] consider an orthogonal problem and use a different technique. [Feng et al., 2026] consider a given and known stochastic system model, and use observation-aware barrier functions to update bounds on the safety and reach-avoid probability with respect to that known model. Thus, they are concerned with the problem of refining the probability of satisfying a property in the given model at runtime. On the other hand, our framework is model-free and only takes as input a certificate that has been verified during the design stage. We then use runtime monitoring to detect certificate violations in an unknown deployment model (which may differ from the design model) at runtime. Thus, these are two different and orthogonal problems. We will discuss this in Related Work.
Our main technical contribution is that, to the best of our knowledge, we present the first framework for runtime monitoring and detecting violations in supermartingale certificates for stochastic systems. While our betting strategies build upon prior work, their application to runtime certificates is novel. Moreover, as discussed in l.112-118, our framework is applicable to all linear temporal logic and omega-regular properties, which is a significantly different and novel domain to which betting strategies have previously not been applied.
[Q2] We distinguish parameters relevant for soundness (l.287-300) and for detection power (l.302-314). We will clarify this and provide full parameter details in the final version:
Soundness: We choose the conventional false-alarm tolerance of 0.05. The loss bounds follow from certificate nonnegativity or known one-step bounds from the dynamics. We use the exact nominal drift for Hill, Ring and LQR. For Building and Tanks, we use independently verified upper bounds that retain some slack. For neural certificates, we test nonincrease and two decrease requirements motivated by the training parameter and reduced step size; these conditions are not verified for the adapted dynamics.
Power: Spatial and temporal scales affect detection power: small scales isolate local violations, while larger scales use more observations and reduce noise. For synthetic benchmarks, we calibrate scales on independent reference states or observation ages to target effective sample sizes from 32 to 8192. For control and neural benchmarks, we equally mix five scales spanning short to long histories and small to broad spatial neighbourhoods. These choices remain fixed for all benchmarks, to avoid tuning the monitor to a particular benchmark.
[Q3] We clarify that, for neural experiments, it is not possible to evaluate the ground truth for the expected value since there is no closed-form expression for the expected value of a neural network function. We will add a sentence to clarify this.
Regarding conservatism of the approach, the most significant source of conservatism is that when we bound the allowed drift, we may subtract more than necessary from the observed certificate change. In principle, this may make the certificate change harder to detect and require more observations than necessary. But we did not observe this to be an issue in our experiments. We will add this discussion.