- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Normalizing Flow Filter (NFF) for non-Gaussian data assimilation using observations alone. The method builds on existing variational filtering approaches, using two normalizing flows to represent the filtering and backward distributions with a fixed observation window. The main extension is an additional future-observation CRPS loss, which improves practical training without changing the theoretical global optima under the stated assumptions. The paper also provides theoretical analysis and experiments on systems up to 16,384 dimensions.
Critical strengths
- Interesting insight into the complementary roles of ELBO and CRPS. The paper shows that good marginal predictive accuracy does not necessarily mean that the learned joint filtering distribution is accurate. The experiments demonstrate that CRPS alone can achieve good marginal scores but poor joint density estimates. Combining ELBO and CRPS helps address both aspects. I find this an interesting and useful insight for probabilistic DA.
- Solid theoretical analysis and well-designed experiments. The theoretical analysis is relatively complete, including the expected ELBO gap, posterior recovery guarantees, and finite-window approximation. The experiments are also carefully designed. In particular, the supplementary-observation experiments provide useful evidence about the quality of the learned joint distribution, beyond standard marginal metrics.
Critical weaknesses
- The motivation and methodological novelty are not clearly presented. The Introduction highlights four DA challenges, but several have already been addressed by prior work. The two-flow formulation and recursive ELBO are also closely related to existing methods. I think the most interesting contribution is actually the combination of ELBO and CRPS, especially the distinction between marginal predictive accuracy and joint posterior quality. However, this motivation is not clearly developed. Instead, the Method section focuses heavily on mathematical derivations, making it difficult to identify the main new idea and why it matters.
- The scalability claim is not sufficiently supported. Although the 16,384-dimensional Kolmogorov-flow experiment is useful, this is still much smaller than realistic atmospheric DA systems, which may involve millions of state variables. The paper also lacks a systematic analysis of how training cost, memory usage, and accuracy scale with state dimension. It remains unclear whether the method can handle much larger systems, especially given the need to train two high-dimensional normalizing flows.
Questions for authors
- What is the main methodological novelty compared with existing variational and flow-based filters? Would it be more appropriate to focus the motivation on the complementary roles of ELBO and CRPS, rather than the four broad DA challenges?
- Why does adding CRPS improve training so significantly? Is this mainly due to better mode coverage or optimization stability? Could the authors provide more analysis or controlled ablations to explain this effect?
- How well does NFF scale to larger systems? Could the authors provide scaling results or discuss the main challenges of applying it to realistic atmospheric DA?
2 comments
Thank you for the detailed response. I appreciate the authors' efforts in addressing my concerns. I think the clarification of the methodological contribution and the additional analysis of CRPS have helped me better understand the motivation and the benefits of the proposed approach. In particular, the explanation of mode coverage and training stability is useful.
Although I still have some concerns about the methodological novelty, as several components build on existing methods, (scalability to realistic atmospheric DA systems remains an open question as well), I do agree that the complementary roles of ELBO and CRPS provide valuable insights into probabilistic data assimilation, and the experimental results are convincing.
Overall, the response has addressed most of my concerns, and I have decided to raise my score from 2 to 3.
Thank you for identifying the distinction between marginal predictive accuracy and joint filtering quality as the most informative part of the paper. We agree that this distinction should appear earlier. We will use the four requirements to define the scientific problem and introduce the complementary roles of the two objectives before presenting the derivations.
1. What is the methodological contribution? We build on the AELBO-2 factorization and on prior work using filtering and backward flows. Our contribution is a method that learns an explicit, potentially multimodal filtering density over a high-dimensional physical state from observations alone, while calibrating the prescribed dynamical and observation models. To our knowledge, NFF is the first amortized filter demonstrated to meet all these requirements together.
This setting poses a specific learning problem. The data let us check whether a filter predicts future observations accurately, but good predictions for individual observed coordinates can coexist with a poor joint density over the latent state. NFF therefore uses future-observation CRPS to score the observable consequences of the filter’s uncertainty, and the recursive ELBO to constrain the joint distribution of successive states through the prescribed model. The supplementary-observation experiment illustrates why this matters: on Lorenz–96, the CRPS-only flow slightly outperformed full NFF in marginal state CRPS (0.823 versus 0.829), yet after assimilating ten additional signed measurements, full NFF gave better predictions at the thirty coordinates those sensors did not measure (0.814 versus 0.876). We will use this example to motivate the method.
2. Does CRPS improve mode coverage or optimization stability? We see evidence for both. In the nonlinear Lorenz–96 experiment, a magnitude observation gives opposite signs the same likelihood. We observe that an ELBO-only filter can concentrate prematurely on one sign. Because that filter supplies the prior for the next update, the underweighted branch can remain difficult to recover.
A simple calculation explains what future observations add. Suppose the two signs imply future observation CDFs G+ and G− for one coordinate, and the only forecast error is the positive-branch probability (w predicted, p true). The excess expected CRPS is (w−p)2∫(G+(z)−G−(z))2dz. Thus a future observation penalizes an earlier sign-probability error precisely when the two signs imply distinguishable future measurements. The exact filter already minimizes this term. In a new two-seed weight sweep, increasing the CRPS weight from 3 to 10 to 30 improved the late-time sign Brier score from 0.1337 to 0.1010 to 0.0996, directly measuring the quality of those probabilities.
The linear-observation control gives a different result. With signed observations, the ELBO-only and combined flows achieved similar state CRPS among successful runs (0.445 and 0.449), but only 3/5 ELBO-only runs succeeded, compared with 5/5 combined runs. This suggests a benefit to training reliability even when the measurements reveal the sign. The experiments support a role for both effects, although they do not quantify their relative contributions.
3. How far does the scalability evidence extend? At 16,384 state variables on a 128 × 128 grid, training the full model for 80,000 updates took 30 hours on one NVIDIA GH200. The backward flow is used during training. Online filtering uses the observation encoder and filtering flow alone: generating 128 posterior samples for one trajectory took 331 ms, including observation encoding, with samples generated in chunks of eight (Appendix Tables 5 and 7).
We agree that this establishes feasibility at the tested resolution and leaves scaling to millions of variables open. At fixed width, depth, and window length, the convolutional flow layers share weights across grid cells, but their activation memory grows with grid size. Training also requires differentiating through simulator rollouts. Finer grids may require shorter solver steps, and larger atmospheric domains may require modeling dependencies over longer distances. We will state the demonstrated scale explicitly and discuss these costs. A controlled resolution sweep measuring training time, peak memory, and filtering accuracy would be needed to support extrapolation to operational atmospheric DA.
Would this revised framing, the sign-probability analysis, and the narrower scalability claim address your concerns sufficiently for you to reconsider your assessment?