- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper introduces a novel mechanism for monitoring formally specified properties in partially observable stochastic systems. The approach is zero-shot: it learns a general representation that can be used to monitor any property at test time without retraining. Instead of predicting full future state sequences, the method predicts only the probability of future words, and employs a factorised representation to circumvent the exponential complexity. This factorisation relies on a linearity assumption of the word probabilities. Given a new specification at test time, the proposed method uses formula progression to precompile a single vector representation thereof (based on the bounded formula horizon), which is then combined linearly with the learned observation history representation to predict the satisfaction probability. Finally, the paper employs histogram binning to provide a PAC-style certificate for the satisfaction probability.
Critical strengths
- The paper proposes an interesting method for a well-motivated problem with applicability in a variety of domains.
- The problem statement and the desirability of fixed-size, reusable representations is clearly developed and motivated.
- Using future words as tests is a natural idea that greatly simplifies the monitoring problem. The authors acknowledge the exponential complexity of this approach in general, and propose a scalable factorisation technique to circumvent this.
- The experiments demonstrate that the proposed method retains much of the performance of the per-property reference, while being general and applicable to arbitrary properties without further training. Compared to sampling-based approaches, the method only needs to compute a single dot product for decoding, leading to much faster inference at some performance cost.
Critical weaknesses
- The assumption of linearity (i.e. the span assumption) seems somewhat strong and is not adequately discussed in the main text. When can this reasonably be assumed to hold in practice? If the assumption does not hold, does the proposed method still recover some approximate solution? This is especially relevant since the approach relies on factorising "all the way" to single letters. Some further analysis or discussion of this assumption would strengthen the paper.
- The main advantage of the proposed factorisation is to circumvent the exponential complexity in the formula horizon. However, as the authors acknowledge in Section 5.1, the long-horizon accuracy of the factorisation significantly degrades due to errors accumulating. Some more systematic analysis of the prediction accuracy w.r.t. formula horizon, and the impact of factorisation for formulae with small horizon would be interesting.
- The exposition reads a bit dense (although this might be because I'm not too familiar with the topic). Would it be possible to describe the factorisation more intuitively, and potentially include an example? Similarly, I found the architecture overview in the appendix quite helpful and encourage the authors to include this in the main paper. A real-world example of an STL specification and how monitoring it is useful could also help to make the background section easier to understand.
- The certification procedure seems somewhat tacked on and separate from the main proposed method. As far as I can tell, this generally certifies the satisfaction of a formula based just on the trajectories - independent of the predictive state representation - and mainly consists in applying the histogram binning bounds from Gupta et al.
- While the paper demonstrates the methods on three different experiments, they all stem from the same domain of vehicle driving. This is representative and includes real-world data (the Perception dataset), but experiments on further domain would make the paper stronger. It would also be appreciated if the authors could report standard deviations or confidence intervals, especially for the timing experiments.
- To add to the above, the authors filter out candidate properties from the experiments based on how well the baseline per-property reference performs compared to a constant baseline. While this seems sensible, it leads to only 4 properties remaining for the Perception benchmark, which is the only benchmark where the proposed method outperforms the RNN and AR-64. Could this be due to the structure of these properties? Can the authors report the numbers for the full set of properties?
Questions for authors
- Can you provide numbers for the full set of properties for the Perception benchmark?
- Can you provide a systematic analysis of the scaling behaviour of the proposed method?
3 comments
Thank you for the detailed rebuttal. I appreciate the updates to the paper, especially regarding exposition, the span assumption, and the additional results.
Certification. I agree that the certificate is useful within the framework. However, it remains a generic component that can be applied to other methods, and there is little novelty in it. Further, it is limited to a fixed property and may be uninformative in practice.
Domain. I appreciate that driving is a demanding domain with complex scenes. I still think that at least one further, potentially significantly different, domain would strengthen the paper.
While these issues remain, I think the rebuttal has improved the paper, and I have hence decided to update my score to 3.
Additional Intensive Care Benchmark
We thank the reviewer for the reassessment and the increased score.
To address their request for a substantially different domain, we ran our monitor on intensive care data, where monitoring anticipates health deterioration that calls for clinical attention. We split data by patient, so the test patients are unseen. The monitor is trained across patients and conditions each prediction on the current patient's history. As in the paper, we compare our zero-shot monitor with a reference trained separately for each property.
We use the PhysioNet/CinC Challenge 2019 data of 40,336 ICU patients (one time step = one hour). The observations are vital signs, laboratory values, and demographics. The atomic propositions include tachycardia (heart rate above 100 beats per minute), hypotension, and fever. We monitor properties over horizons of 3, 6, and 12 hours, e.g., "hypotension within 6 h", "sustained hypotension within 12 h", and "no hypotension until fever". Over three random patient splits, our monitor reaches 95% of the reference's skill (0.951 ± 0.004), with AUROC 0.872 against 0.881:
| horizon | 3 h | 6 h | 12 h |
|---|---|---|---|
| skill relative to the reference | 0.997 | 0.972 | 0.899 |
Reply to Reviewer VPiq
We thank the reviewer for the detailed feedback and for recognizing the submission's strengths and potential.
Span assumption and factorization (W1, W2). As explained in the general comment (point 1), Assumption 1 is needed only for exact decoding (Theorem 1). Otherwise, the monitor approximates the satisfaction probability (Sec. 4.1), and the certificate remains valid (Proposition 1). Factorizing down to single letters has little accuracy cost at short horizons, with skill relative to the reference of 0.99 at H=5 (App. I.2). We now quantify the long-horizon loss, including the contribution from lost probability mass (Sec. 5 and App. I.2). Beyond making runtime independent of H, factorization allows compilation at any horizon without enlarging the core-test set or retraining.
Exposition (W3). We followed the reviewer's suggestions. The extra page allowed us to move the architecture overview into Section 4. Section 2.2 now illustrates the setting with a benchmark property for sustained tailgating, □[0,36]◊[0,3]ph. It requires the headway to the vehicle ahead to recover to at least two seconds at least once in every four consecutive timesteps over the next 40 steps.
Certification (W4). The certificate is necessary for the monitor's correctness, rather than an optional addition. Predictions are exact when Assumption 1 holds and the predictive state and operators are known exactly. Both are estimated from finite data, so a raw prediction for a new property has estimation error of unknown size. The certificate bounds the bin-conditional satisfaction probability of each property, separately, regardless of that error, providing the guarantee needed for safety decisions. We use the histogram binning of Gupta et al. without modification and do not claim a new calibration method. Its independence from the representation is deliberate, so its guarantee does not depend on Assumption 1. Applicability to other monitors does not remove the need to certify ours. For zero-shot monitoring, a new property requires no new trajectories, inference, or training for recalibration, only stored labels (0.15–1.4 ms).
Domain and uncertainty (W5). Our limitations acknowledge that the evidence covers one application domain. Driving is nevertheless a demanding test case that is representative of real-world challenges in other domains. The datasets are real, and the scenes involve complex multi-agent interactions, with the monitored vehicle's future depending on many other road users. The three benchmarks span the perception stack, from agent tracks through bird's-eye-view renderings to raw camera frames. The revision adds seed standard deviations to Table 1 and interquartile ranges to all timings.
Perception properties (W6, Q1). Table 3 lists every candidate property, and the revision adds every monitor's Brier score for the excluded properties (general comment, point 2). Our advantage over the RNN and AR-64 persists across all 33 candidate Perception properties with both outcomes on the test split: our mean test Brier score is 0.0750, compared with 0.0778 for the RNN and 0.0851 for AR-64. Most were excluded because the frozen DINOv2 features carry little information about them. Training a dedicated encoder requires more vision data and compute than we had.
Scaling (Q2). Online cost is independent of the horizon and grows only slightly with the number of properties (Appendices I.2 and I.3). Exponential scaling arises only with the number of propositions.