acceptodds
Under review as a conference paper at ICLR 2027

Normalizing Flow Filter: Observation-Only Learning for Non-Gaussian Data Assimilation

Abstract

State filtering is central to data assimilation and model-based probabilistic forecasting in high-dimensional dynamical systems. Many scientific applications require filtering methods that can simultaneously: (i) represent non-Gaussian posteriors, (ii) scale to high dimensions, (iii) train from observations alone, and (iv) retain explicit transition and observation models for calibrating physical parameters and sensor biases. To directly address this setting, we introduce the Normalizing Flow Filter (NFF), an amortized filter with -step memory that represents the adjacent-state posterior using two conditional normalizing flows. The training objective combines a recursive filtering evidence lower bound with the continuous ranked probability score on short-horizon future observations. For a fixed, correctly specified state-space model including the initial-state prior, exact -step memory, and sufficient flow expressivity, every global minimizer of the NFF loss averaged over data-generating observation sequences recovers the exact filtering posterior. NFF closely matches a -particle-filter reference for a four-dimensional coupled oscillator, remains reliable under highly ambiguous nonlinear observations in 40-dimensional Lorenz–96, and scales to joint state, physical-parameter, and sensor-bias inference in 16,384-dimensional Kolmogorov flow.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
data assimilation, non-Gaussian posterior, state filtering, variational inference

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ZXHvOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Normalizing Flow Filter (NFF) for non-Gaussian data assimilation using observations alone. The method builds on existing variational filtering approaches, using two normalizing flows to represent the filtering and backward distributions with a fixed observation window. The main extension is an additional future-observation CRPS loss, which improves practical training without changing the theoretical global optima under the stated assumptions. The paper also provides theoretical analysis and experiments on systems up to 16,384 dimensions.

Critical strengths
  1. Interesting insight into the complementary roles of ELBO and CRPS. The paper shows that good marginal predictive accuracy does not necessarily mean that the learned joint filtering distribution is accurate. The experiments demonstrate that CRPS alone can achieve good marginal scores but poor joint density estimates. Combining ELBO and CRPS helps address both aspects. I find this an interesting and useful insight for probabilistic DA.
  2. Solid theoretical analysis and well-designed experiments. The theoretical analysis is relatively complete, including the expected ELBO gap, posterior recovery guarantees, and finite-window approximation. The experiments are also carefully designed. In particular, the supplementary-observation experiments provide useful evidence about the quality of the learned joint distribution, beyond standard marginal metrics.
Critical weaknesses
  1. The motivation and methodological novelty are not clearly presented. The Introduction highlights four DA challenges, but several have already been addressed by prior work. The two-flow formulation and recursive ELBO are also closely related to existing methods. I think the most interesting contribution is actually the combination of ELBO and CRPS, especially the distinction between marginal predictive accuracy and joint posterior quality. However, this motivation is not clearly developed. Instead, the Method section focuses heavily on mathematical derivations, making it difficult to identify the main new idea and why it matters.
  2. The scalability claim is not sufficiently supported. Although the 16,384-dimensional Kolmogorov-flow experiment is useful, this is still much smaller than realistic atmospheric DA systems, which may involve millions of state variables. The paper also lacks a systematic analysis of how training cost, memory usage, and accuracy scale with state dimension. It remains unclear whether the method can handle much larger systems, especially given the need to train two high-dimensional normalizing flows.
Questions for authors
  1. What is the main methodological novelty compared with existing variational and flow-based filters? Would it be more appropriate to focus the motivation on the complementary roles of ELBO and CRPS, rather than the four broad DA challenges?
  2. Why does adding CRPS improve training so significantly? Is this mainly due to better mode coverage or optimization stability? Could the authors provide more analysis or controlled ablations to explain this effect?
  3. How well does NFF scale to larger systems? Could the authors provide scaling results or discuss the main challenges of applying it to realistic atmospheric DA?
2 comments

Thank you for identifying the distinction between marginal predictive accuracy and joint filtering quality as the most informative part of the paper. We agree that this distinction should appear earlier. We will use the four requirements to define the scientific problem and introduce the complementary roles of the two objectives before presenting the derivations.

1. What is the methodological contribution? We build on the AELBO-2 factorization and on prior work using filtering and backward flows. Our contribution is a method that learns an explicit, potentially multimodal filtering density over a high-dimensional physical state from observations alone, while calibrating the prescribed dynamical and observation models. To our knowledge, NFF is the first amortized filter demonstrated to meet all these requirements together.

This setting poses a specific learning problem. The data let us check whether a filter predicts future observations accurately, but good predictions for individual observed coordinates can coexist with a poor joint density over the latent state. NFF therefore uses future-observation CRPS to score the observable consequences of the filter’s uncertainty, and the recursive ELBO to constrain the joint distribution of successive states through the prescribed model. The supplementary-observation experiment illustrates why this matters: on Lorenz–96, the CRPS-only flow slightly outperformed full NFF in marginal state CRPS (0.823 versus 0.829), yet after assimilating ten additional signed measurements, full NFF gave better predictions at the thirty coordinates those sensors did not measure (0.814 versus 0.876). We will use this example to motivate the method.

2. Does CRPS improve mode coverage or optimization stability? We see evidence for both. In the nonlinear Lorenz–96 experiment, a magnitude observation gives opposite signs the same likelihood. We observe that an ELBO-only filter can concentrate prematurely on one sign. Because that filter supplies the prior for the next update, the underweighted branch can remain difficult to recover.

A simple calculation explains what future observations add. Suppose the two signs imply future observation CDFs G+G_+ and G−G_- for one coordinate, and the only forecast error is the positive-branch probability (ww predicted, pp true). The excess expected CRPS is (w−p)2∫(G+(z)−G−(z))2 dz.(w-p)^2 \int (G_+(z)-G_-(z))^2\,dz. Thus a future observation penalizes an earlier sign-probability error precisely when the two signs imply distinguishable future measurements. The exact filter already minimizes this term. In a new two-seed weight sweep, increasing the CRPS weight from 3 to 10 to 30 improved the late-time sign Brier score from 0.1337 to 0.1010 to 0.0996, directly measuring the quality of those probabilities.

The linear-observation control gives a different result. With signed observations, the ELBO-only and combined flows achieved similar state CRPS among successful runs (0.445 and 0.449), but only 3/5 ELBO-only runs succeeded, compared with 5/5 combined runs. This suggests a benefit to training reliability even when the measurements reveal the sign. The experiments support a role for both effects, although they do not quantify their relative contributions.

3. How far does the scalability evidence extend? At 16,384 state variables on a 128 × 128 grid, training the full model for 80,000 updates took 30 hours on one NVIDIA GH200. The backward flow is used during training. Online filtering uses the observation encoder and filtering flow alone: generating 128 posterior samples for one trajectory took 331 ms, including observation encoding, with samples generated in chunks of eight (Appendix Tables 5 and 7).

We agree that this establishes feasibility at the tested resolution and leaves scaling to millions of variables open. At fixed width, depth, and window length, the convolutional flow layers share weights across grid cells, but their activation memory grows with grid size. Training also requires differentiating through simulator rollouts. Finer grids may require shorter solver steps, and larger atmospheric domains may require modeling dependencies over longer distances. We will state the demonstrated scale explicitly and discuss these costs. A controlled resolution sweep measuring training time, peak memory, and filtering accuracy would be needed to support extrapolation to operational atmospheric DA.

Would this revised framing, the sign-probability analysis, and the narrower scalability claim address your concerns sufficiently for you to reconsider your assessment?

Thank you for the detailed response. I appreciate the authors' efforts in addressing my concerns. I think the clarification of the methodological contribution and the additional analysis of CRPS have helped me better understand the motivation and the benefits of the proposed approach. In particular, the explanation of mode coverage and training stability is useful.

Although I still have some concerns about the methodological novelty, as several components build on existing methods, (scalability to realistic atmospheric DA systems remains an open question as well), I do agree that the complementary roles of ELBO and CRPS provide valuable insights into probabilistic data assimilation, and the experimental results are convincing.

Overall, the response has addressed most of my concerns, and I have decided to raise my score from 2 to 3.

Reviewer_BKkxOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

The paper addresses the problem of estimating the filtering distribution in a first-order Markov state-space model. The paper trains the AELBO-2 objective (see Campbell et al. (2021)) plus a scoring rule (predictive CRPS), with filtering distribution parametrized as normalizing flow. The paper compares the proposed method against some previous methods on a few systems, including nonlinear Lorenz–96 filtering and 16,384-dimensional Kolmogorov flow.

Critical strengths

The exact combination of a predictive CRPS term and normalizing flow parametrization with AELBO-2 objective seem to be new and interesting for learning in this context.

Critical weaknesses

The paper combines a number of known techniques, in a slightly straightforward way. This could result in a significant contribution if the authors could systematically show that this precise combination of objective and parametrization leads to superior results than existing methods in terms of computation/performance tradeoff. Th methods are compared to a few out of the many methods reviewed in the Related Works which are appropriate to compare against. It will be great to expand the experiments in the paper.

Questions for authors

Please see the critical weaknesses.

1 comment

Thank you for recognizing the novelty of the proposed combination. Our main contribution is a filtering method that addresses four requirements together: representing potentially multimodal filtering distributions, scaling to high-dimensional physical states, learning from observations alone, and retaining the prescribed dynamical and observation models so their parameters can be calibrated from those same observations. To our knowledge, NFF is the first amortized filter demonstrated to meet all four requirements with an explicit density over the current physical state. We believe that describing the method solely in terms of its familiar components understates this contribution.

Why do these requirements matter together? In our Kolmogorov experiment, some sensors measure squared vorticity, so the same reading is compatible with opposite signs. The forcing amplitude, Reynolds number, and sensor biases are also unknown. Generating supervised state trajectories would require choosing values or a sampling distribution for the very parameters we seek to estimate. NFF uses the prescribed physics and the observed sequences to learn a filtering density over 16,384 state variables while jointly calibrating those parameters.

What does the combination contribute? NFF builds on the AELBO-2 factorization. The recursive ELBO constrains the joint distribution of successive states through the prescribed model. Future-observation CRPS evaluates whether samples from the filter, propagated forward, predict what the sensors later observe. For a fixed, correctly specified model, we give sufficient conditions under which every global minimizer of the combined population objective recovers the exact filter.

The five-seed Kolmogorov ablation illustrates why both objectives matter. The CRPS-only flow estimated the forcing and sensor biases accurately, yet its state CRPS was 2.068. Adding the recursive ELBO reduced state CRPS to 0.231. The ELBO-only flow scored 2.639, and replacing the flow with a diagonal Gaussian under the combined objective gave 2.404. Thus the CRPS-only result reveals a failure that parameter estimates alone would have missed: a model can calibrate the simulator accurately while learning a poor filtering distribution.

New accuracy–computation comparison. We added PR-Smoother and EnFF on nonlinear Lorenz–96. Each method received the same first 16 observations and targeted the same endpoint distribution, where filtering and smoothing coincide. We report two-seed means, with 512 held-out trajectories per seed and 128 members per trajectory. Lower CRPS is better.

MethodTraining (GPU-h/seed)Endpoint state CRPSEndpoint latency (ms)
PR-Smoother (359,000 updates)20.074.18341.09
EnFF02.24143.64
NFF (λ = 3)19.111.11110.79
NFF (λ = 10, paper default)—0.82210.32
NFF (λ = 30)19.610.80510.02

Both NFF configurations with measured training times used slightly fewer GPU-hours than PR-Smoother, achieved 73–81% lower endpoint CRPS, and produced the endpoint ensemble in about one quarter of the time. A separate training time is unavailable for the default checkpoint. EnFF offers a different tradeoff during sustained filtering: over steps 257–512, each EnFF update took 2.77 ms versus 9.91 ms for default NFF, with state CRPS of 1.986 versus 0.825.

We will revise the introduction to foreground the joint filtering and calibration problem and bring the accuracy–computation comparison into the main text. Given these new comparisons and the evidence for the contribution of each component, would you reconsider your assessment of the paper’s significance and your rating?

Reviewer_zCKzOpenReview ↗
Rating
3
Confidence
2(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes the Normalizing Flow Filter (NFF), an amortized variational filter for non-Gaussian data assimilation that learns from observations alone. First, the authors propose a filtering flow and a backward flow that are trained with a recursive filtering ELBO while calibrating the prescribed model parameters. Then, a future-observation CRPS is added to improve mode coverage, and the authors show that the global optima of the combined objective recover the exact filter. Evaluated on a coupled oscillator, Lorenz–96, and high-dimensional Kolmogorov flow, the proposed method demonstrates clear improvements in filtering accuracy and joint parameter inference. The authors also provide ablation studies and sensitivity analysis to further support the method.

Critical strengths
  • Clear Motivation: The paper presents four requirements for data assimilation, non-Gaussian posteriors, high-dimensional scalability, observation-only learning, and explicit model calibration. The authors then present a single framework to address the four requirements.

  • Theoretical Grounding: The training objective comes with an exact expected-gap decomposition into a predictive and a posterior KL terms, which leads to exact filter recovery at global optima. The authors further show that adding the future-observation CRPS preserves these global optima, leaving the theoretical target unchanged.

  • Comprehensive Ablation Studies: The ablations isolate each loss term and the posterior family, showing that strong marginal scores alone can be misleading. For instance, the result in Table 1 shows that CRPS-only variant achieves good state CRPS but a much worse filtering NLL.

Critical weaknesses
  • Comparison with Related Methods: The only learned baseline is AmEnF, while several methods discussed in the related work, including observation-only variational approaches such as AELBO-1 and PR-Smoother and the training-free EnFF, are not compared. Comparing with these methods would clarify both the benefit of the proposed design and the cost of learning without state supervision.

  • Justification for CRPS: While ablation studies demonstrate that training with ELBO alone fails in the presence of nonlinear observations, the paper only presents these empirical results without analyzing the underlying reasons for the failure. Given that the theoretical analysis only guarantees that adding CRPS preserves global optima, the paper would be strengthened by including a conceptual explanation or analytical discussion on why CRPS is necessary in nonlinear regimes.

Questions for authors
  • Please add comparison with the more related works such PR-Smoother, AELBO-1 and EnFF. Specifically, How does NFF compare with PR-Smoother at the final time step, where the smoothing and filtering distributions coincide, given that both methods operate in the same observation-only prescribed-model setting?
  • Since the theory implies that CRPS does not change the global optima, could the authors provide any discussion or analysis to justify the necessity of CRPS?
  • How sensitive are the results to λCRPS\lambda_{CRPS} ?
1 comment

Thank you for suggesting an endpoint comparison, where smoothing and filtering target the same distribution.

1. Comparison with related methods.

We evaluated 40-dimensional Lorenz--96 with noisy magnitude observations, yt=∣xt∣+ϵty_t=|x_t|+\epsilon_t. Each method received the same first 16 observations and estimated p(x16∣y1:16)p(x_{16}\mid y_{1:16}). NFF and PR-Smoother trained on observations alone with the prescribed model. PR-Smoother trained for 359{,}000 updates (20.07 GPU-hours per seed), comparable to our NFF runs at λ=3,30\lambda=3,30 (19.11 and 19.61 hours). EnFF requires no training.

MethodState CRPSCalib. error (pp)Sign BrierEndpoint (ms)
NFF (λ = 10)0.822 ± 0.0121.220.10210.32
PR-Smoother4.183 ± 0.03027.630.45241.09
EnFF2.241 ± 0.0096.220.23543.64

Values are means over two seeds, with 512 held-out trajectories per seed and 128 members per trajectory. ±\pm gives the across-seed SD. Lower is better throughout. NFF gave the most accurate sign probabilities at the endpoint. EnFF was faster per subsequent analysis cycle (2.77 vs.\ 9.91 ms for NFF over steps 257--512). These comparisons measure the methods' performance under observation-only access. Isolating the cost of withholding state labels would require a supervised control.

A difference from PR-smoother.

Consider one Lorenz--96 coordinate with forecast N(0,s2)N(0,s^2) given a fixed predecessor. After a magnitude observation yt=a>0y_t=a>0 with Gaussian noise of variance r2r^2, the posterior has two modes at ±as2/(s2+r2)\pm a s^2/(s^2+r^2), near −a-a and +a+a when r≪sr\ll s. Both signs are plausible continuations of the same state. PR-Smoother uses one Gaussian forward conditional for each predecessor and observation sequence, so that conditional cannot represent both peaks. Its marginal can still become bimodal by mixing over predecessors. NFF fits the filtering marginal directly with a flow at each step, allowing it to represent a single peak at one step and two separated sign modes at the next.

2. Why does future-observation CRPS help?

In ELBO-only runs, we observe that a filtering marginal sometimes concentrates on one sign at an intermediate step and the other sign remains underrepresented later. The local ELBO is

Lt=log⁡p~t(yt∣Wt−1)−KL(qt∥π~t).\mathcal L_t=\log\tilde p_t(y_t\mid W_{t-1}) -\mathrm{KL}(q_t\|\tilde\pi_t).

The first term scores the one-step forecast from the preceding observation window Wt−1W_{t-1}. The reverse KL fits the adjacent-state posterior qtq_t to its target π~t\tilde\pi_t and can favor one of two separated modes. The next target inherits the learned filter qtFq^F_t:

π~t+1(xt,xt+1)∝qtF(xt∣Wt)pθ(xt+1∣xt) pθ(yt+1∣xt+1).\tilde\pi_{t+1}(x_t,x_{t+1})\propto q^F_t(x_t\mid W_t) p_\theta(x_{t+1}\mid x_t)\,p_\theta(y_{t+1}\mid x_{t+1}).

If one branch receives little mass, its continuations start with little weight. The current magnitude observation gives opposite signs the same likelihood.

CRPS scores rollouts from time tt against future observations during training. Suppose the two signs induce future observation CDFs G+G_+ and G−G_- for one coordinate. If the only error is the positive-branch probability (ww predicted, pp true), the excess expected CRPS is

ΔCRPS=(w−p)2∫R(G+(z)−G−(z))2 dz.\Delta_{\rm CRPS}=(w-p)^2\int_{\mathbb R}(G_+(z)-G_-(z))^2\,dz.

Thus a later observation penalizes an earlier branch-weight error when the branches predict different magnitudes. Like forward KL, CRPS scores forecasts against outcomes drawn from the data. Its divergence is the squared CDF distance above. The penalty vanishes at w=pw=p, preserving the exact population optimum under our assumptions. In the five-seed Lorenz--96 ablation, adding CRPS to the same flow reduced state CRPS from 3.352 to 0.829 and filtering NLL per dimension from 15.672 to 1.602.

3. Sensitivity to λCRPS\lambda_{\rm CRPS}.

We varied the weight and averaged over steps 257--512 and two seeds (lower is better).

CRPS weightState CRPSSign BrierFilter NLL/dim.
31.1000.13371.872
100.8250.10101.598
300.8130.09961.755

Increasing the weight from 3 to 10 improved all three scores. At 30, state CRPS and sign Brier improved slightly further, while the joint density score worsened. We therefore retain 10 as the default.

Comments on the paper

1 comment

Dear authors and reviewers,

The author-reviewer discussion window will close tomorrow on November 18.

@authors: If not done already, please respond to the reviewers' comments before the discussion window closes. I would advise short and concise responses over lengthy explanations.

@reviewers: Please check the authors' responses and provide any additional comments before the discussion window closes. Consider updating your initial review if necessary, and in particular your score if your opinion has changed. You may increase your score if you feel the authors' responses warrant it, maintain your score if your opinion remains the same, or even decrease your score if you feel the other reviews or the authors' responses reveal new shortcomings. At the very least, please acknowledge the authors' responses.

Thank you for your attention and cooperation.

The AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.