- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper studies the question of estimating a dynamical system in which (1) the dynamics are unknown, (2) the measurement model on clean data is known, (3) some unknown fraction of the data is arbitrarily corrupted and carries no information about the underlying state. In the absence of a model for the dynamics, it is challenging to identify the corruptions, and prior work in this literature either had to assume a model for the dynamics or were not robust. This paper proposes the following variational method: define indicator variables for the corrupted steps with some manually chosen prior over these variables, formulate an autoregressive parametrization for the unknown dynamics as well as a specific parametrization for the posterior on the state given the measurements and the corruption indicators. For the latter, this involves a "corruption posterior", which they take to be a GRU approximating the posterior on the corruption variables conditioned on the measurements, and a "state posterior", which they take to be a Gaussian approximating the posterior on the next state conditioned on the previous trajectory and the measurements of the full trajectory apart from the next state. They jointly train these components by minimizing an ELBO loss. Under certain regularity assumptions, for instance coercivity of the network for the corruption posterior, they show that the KL between the model conditioned on corrupted data vs conditioned on clean data is finite.
The main empirical findings are that on synthetic data from a stochastic Lorenz-63 system, their method degrades the most gracefully in the presence of corruptions under linear measurements, and under nonlinear measurements it outperforms other model-free methods though it underperforms relative to a model-based baseline. For human motion capture data, it beats other baselines by a significant margin.
Critical strengths
- The paper studies a fundamental and challenging problem in state estimation and proposes a method that is empirically shown to be more robust than other model-free methods for the same problem.
- The ablations show that having the state posterior at time be independent of the measurement at time is indeed crucial to the performance of their method.
Critical weaknesses
- The theoretical result, which gives a non-quantitative bound under strong assumptions that feels orthogonal to why the model is actually effective empirically, feels a bit like proving a theorem for the sake of proving a theorem and could be safely removed without affecting the overall paper.
- The authors transparently acknowledge various limitations like knowing the measurement model for clean data and assuming independent corruptions across steps. The former feels more in scope for follow-up work , but the latter feels like a more significant shortcoming -- it is relatively unrealistic yet is the basis for the experiments as well, which exclusively involve independent corruptions across steps. While it makes sense to focus on the independent setting in the development of the methods, the paper would be strengthened by augmenting the evaluations with some examples of bursty / correlated corruptions, just to see how robust the method is.
Questions for authors
How does the method perform in settings in which the corruptions are correlated, either across time or with the state itself?
We thank the reviewer for the constructive review and feedback.
W1. We thank the reviewer for this criticism. We agree that the theoretical result is not quantitative, but achieve an asymptotic result of the method, and is the main theorem of the WoLF method (Thm. 3.2, Duran-Martin et al., 2024). They show, and argue, that this theorem is what makes the WoLF method (in its various settings) asymptotically outlier-robust. And thus, we argue that our main proposition is what makes us call RoVaS outlier-robust, as the proposition shows that it is (asymptotically) outlier-robust. We hope this gives some insight into our thought process on the addition of the proposition. For an estimation of the KL in the proposition, we refer to Fig. 4, where we show RoVaS achieving magnitudes lower KL than VaS and VaS-t.
W2. We thank the reviewer. We have performed correlated outlier experiments. Please see our answer to Q1.
Questions
Q1. We thank the reviewer for the question. To study outlier robustness of RoVaS and the baselines on correlated corruptions, we simulate Lorenz-63 using identity measurement map and bursty outliers using a Markov chain of mean time length 5, expected outlier probability of 10, SMNR 20 dB, and the same hyperparameters as in the manuscript. The results are shown below.
The results show that both RoVaS and VaS-t seem to handle correlated corruptions (bursty corruptions), and the generalization of training on independent outliers and testing on bursty outliers shows reasonable performance, albeit with a slightly larger drop in performance.