acceptodds
Under review as a conference paper at ICLR 2027

LD-EnFF: Latent-Dynamics Ensemble Flow Filtering for Data Assimilation with Sparse Observations

Abstract

Data assimilation combines model forecasts with noisy, incomplete observations to estimate the evolving state of a dynamical system. Existing methods face two compounding challenges: high-dimensional nonlinear dynamics make repeated forward simulation computationally expensive, while sparse observations provide limited direct information about the full state. To address these challenges, we propose the Latent-Dynamics Ensemble Flow Filter (LD-EnFF), a sequential Bayesian filtering framework that performs both forecast propagation and filtering updates in a compact latent space. LD-EnFF combines a latent dynamics surrogate for ensemble propagation with a variational autoencoder (VAE)-based observation model that evaluates a state-dependent observation likelihood in latent space. At each assimilation step, an ensemble filtering update based on flow matching uses the forecast ensemble and this likelihood to generate posterior samples, jointly updating latent states and uncertain parameters. This design avoids repeated full-state simulation during forecasting and full-field reconstruction during likelihood evaluation. LD-EnFF substantially outperforms a broad range of data assimilation algorithms on benchmarks spanning Kolmogorov flow, tsunami propagation, and atmospheric modeling, all featuring complex dynamics and sparse, noisy observations.

Primary area:
learning on time series and dynamical systems
Keywords:
data assimilation, ensemble flow filtering, latent-space filtering, latent dynamics, scientific machine learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_MLYBOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

LD-EnFF performs sequential data assimilation entirely in a learned latent space. An LDNet gives parameter-conditioned latent dynamics. A VAE encodes sparse observations, and an MLP "y⋆y^\star operator" maps the augmented latent state (latent state plus uncertain parameter) to noiseless observations. Together they define a local Gaussian latent likelihood whose covariance combines observation noise pushed through the encoder Jacobian and the encoder variance. Ensemble flow filtering (OT or perturbed F2P paths, Monte Carlo guidance) then updates latent states and parameters jointly. On Kolmogorov flow, tsunami propagation and PlanetSWE, with 0.1–0.44% of grid points observed, LD-EnFF has the lowest final RelRMSE among latent-space methods (e.g. 5.74% vs 13.09% for the matched LSTM-encoder EnFF on Kolmogorov).

Critical strengths
  1. Learning a forward map from latent state to observations, instead of an inverse map from observation histories to states, is a sensible design for sparse data, and it avoids full-state reconstruction during the analysis.
  2. The matched baselines (same LDNet with an LSTM encoder; same likelihood with IEnKF) help locate where the gains come from, and the ranking is consistent across the three benchmarks.
Critical weaknesses
  1. The main new component, the VAE-based latent likelihood, is not tested against the obvious simpler choice. Relative to LD-EnSF, the changes are EnSF → EnFF and the LSTM inverse encoder → VAE plus forward operator, so the likelihood is the key new piece. Since HαH_\alpha already predicts observations, one can use N(y;,Hα(κ),,Γy)\mathcal{N}(y;\\, H_\alpha(\kappa),\\, \Gamma_y) directly, with no encoder, no Jacobian and no reconstruction. This control is missing, and so is any calibration check of the added covariance; only the RelRMSE of the ensemble mean is reported. The comparison with the LSTM encoder changes two things at once: a forward vs an inverse model, and the form of the likelihood.
  2. The benchmarks are easier than the high-dimensional framing suggests. In all three, the initial physical state is shared or known and only one or two parameters are uncertain, and LDNet starts every trajectory from s0=0s_0 = 0, so the task is mostly low-dimensional parameter identification. Figs. 2–3 fit this reading: on Kolmogorov, once the parameter error is down to 1–2%, the LD-EnFF-OT state error keeps growing (about 1% → 5.7%), following the surrogate. The F2P results are also hard to reconcile with the method. With Monte Carlo guidance the guided velocity is a convex combination of conditional velocities, so Eq. 46 with σmin⁡=0.001\sigma_{\min} = 0.001 limits the parameter change to a few thousandths (normalized) per cycle; yet Fig. 3 shows F2P moving Re by several tenths within about 15 cycles. There is no test with uncertain initial fields or model error.
Questions for authors
  1. How does the same LDNet + EnFF perform with the direct likelihood N(y;,Hα(κ),,Γy)\mathcal{N}(y;\\, H_\alpha(\kappa),\\, \Gamma_y)? Please report calibration (CRPS, spread/skill or coverage) for both.
  2. How does F2P achieve the parameter updates in Fig. 3, given Eq. 46 and σmin⁡=0.001\sigma_{\min} = 0.001 (coordinates, integration interval, any inflation or jitter)? Does the implementation differ from Appendix B.2? Under OT, can parameter values leave the set spanned by the initial 20 samples?
  3. How does the method perform when the initial state is perturbed and not fully determined by ε\varepsilon? What is the runtime compared with LETKF, given that efficiency is part of the motivation?
8 comments
AuthorsOpenReview ↗

Response to Reviewer MLYB (1/6)

We thank the reviewer for the careful reading of both the main text and the appendices. Several of the reviewer’s observations were correct and led to substantive revisions, summarized below and in the General Response.

W1. The main new component, the VAE-based latent likelihood, is not tested against the obvious simpler choice. Relative to LD-EnSF, the changes are EnSF → EnFF and the LSTM inverse encoder → VAE plus forward operator, so the likelihood is the key new piece. Since HαH _\alpha already predicts observations, one can use N(y;Hα(κ),Γy)\mathcal N(y; H _\alpha(\kappa), \Gamma _y) directly, with no encoder, no Jacobian and no reconstruction. This control is missing, and so is any calibration check of the added covariance; only the RelRMSE of the ensemble mean is reported. The comparison with the LSTM encoder changes two things at once: a forward vs an inverse model, and the form of the likelihood.

Q1. How does the same LDNet + EnFF perform with the direct likelihood N(y;Hα(κ),Γy)\mathcal N(y; H _\alpha(\kappa), \Gamma _y)? Please report calibration (CRPS, spread/skill or coverage) for both.

AuthorsOpenReview ↗

Response to Reviewer MLYB (2/6)

Response to W1 and Q1. We thank the reviewer for pointing out the missing comparison. We agree that the comparison with the LSTM encoder changes more than one thing at once, including the direction of the observation map (inverse vs. forward) and the form of the likelihood.

Ablation (new Appendix E.3). As suggested, we ran the same LDNet + EnFF-OT with the direct likelihood N(yj;Hα(κ),Γy)\mathcal N(\mathbf y _j;H _\alpha(\boldsymbol\kappa),\boldsymbol\Gamma _y), without the encoder and without the Jacobian, keeping everything else unchanged. This is a controlled comparison of the two likelihoods with the same forward y⋆y^\star operator. As calibration measure we report the spread–skill ratio (SSR) of the reconstructed full state.

BenchmarkLikelihoodFinal RelRMSE (%)SSR
Kolmogorov FlowDirect6.70 ± 0.560.205 ± 0.081
Latent (ours)5.74 ± 0.260.536 ± 0.083
Tsunami ModelingDirect5.99 ± 0.550.398 ± 0.124
Latent (ours)6.53 ± 0.250.701 ± 0.122
Atmospheric ModelingDirect4.80 ± 0.430.113 ± 0.017
Latent (ours)4.42 ± 0.210.232 ± 0.025

The table reports mean ±\pm standard deviation over 20 test trajectories (per trajectory, the RelRMSE is averaged and the SSR pooled over three random seeds). The spread–skill ratio (SSR) measures the ensemble spread relative to the estimation error, with values closer to one indicating better calibration and values below one indicating an overconfident ensemble.

Comparison with the LSTM-based baseline. LDNet+LSTM EnFF-OT shares the LDNet and the EnFF-OT analysis with both variants, but its observation model differs entirely: an inverse LSTM encoder, a likelihood defined in the latent space around the LSTM estimate, and its own covariance. Both forward variants attain lower final errors on all three benchmarks (6.70 / 5.99 / 4.80% and 5.74 / 6.53 / 4.42%, versus 13.09 / 9.87 / 6.18%). Since several components differ, this comparison shows that the observation model built on the forward y⋆y^\star operator outperforms the LSTM-based one, but it does not attribute the gap to a single component, such as the direction of the map or the space in which the likelihood is evaluated. The reviewer’s second factor, the form of the likelihood, is isolated by the controlled comparison above.

Effect of the latent likelihood (controlled comparison). Replacing the direct by the latent likelihood increases the SSR by a factor of 1.8–2.6 on all three benchmarks and roughly halves the variability of the final error across test trajectories. It also lowers the final error on Kolmogorov flow and atmospheric modeling, whereas the direct likelihood is slightly more accurate on tsunami modeling; there it even falls slightly below the oracle surrogate run with the true parameter (Appendix E.2). A likely reason is that selecting the members that best fit each observation partly compensates for surrogate error; this comes at the cost of an overconfident ensemble (SSR 0.40). The revised paper accordingly presents the empirical benefit of the latent likelihood mainly as improved calibration (Section 4.2 and the contributions) and robustness across trajectories (Appendix E.3). The similar accuracy is consistent with our setting: the parameters are time-invariant, the surrogate dynamics are deterministic, and the initial state carries no uncertainty beyond the parameters, so each member’s latent trajectory is determined by its parameter, up to the σmin⁡\sigma _{\min}-scale jitter added at each analysis, and concentrating the ensemble on the best-fitting members mainly degrades calibration rather than accuracy (Appendix E.3). When the forecast is subject to unpredictable state errors, for example from uncertain initial conditions or stochastic model error, we expect this concentration to degrade accuracy as well, which is where the better-conditioned latent likelihood should matter most.

AuthorsOpenReview ↗

Response to Reviewer MLYB (3/6)

Why the latent likelihood helps (new Appendix B.4). Building on the likelihood-weighted resampling characterization of EnFF by Transue et al. (2025), we analyze how observation encoding affects the likelihood weights. Under an affine Gaussian approximation with fixed covariance, we prove that the latent likelihood reduces the observation-averaged variance of the log-likelihood weights relative to the direct physical likelihood (Proposition 1). We further establish a corresponding bound on the expected dispersion of normalized log-weights for any finite ensemble size (Corollary 2). These results provide a theoretical mechanism by which observation encoding can mitigate weight concentration in small ensembles, without guaranteeing larger effective sample size or improved filtering accuracy.

Calibration. Both ensembles remain underdispersed (SSR below one). For the latent likelihood, we identify several reasons, now discussed in Appendix E.3: late in the window, the full-state error is dominated by surrogate error that is largely common to all members (Appendix E.2) and is therefore only partly reflected in the ensemble spread; the selection-based analysis gradually reduces ensemble diversity (Corollary 5); and the approximation error of HαH _\alpha is not modeled (Appendix D.2). Covariance inflation or an explicit model-error term would further improve calibration, which we leave to future work.

Reviewer_54TnOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The work builds on previous works, namely Latent EnSF and LD-EnSF, which adapts the EnSF framework to learned states and observations in a latent space. Aside from using EnFF with MC guidance instead of EnSF as the filtering backbone, the main contribution of the work is in how observations are handled: while Latent EnSF and LD-EnSF learn a mapping from observations to a latent representation of the corresponding ground truth, a fundamentally ill-posed map, the proposed LD-EnFF instead learns a latent representation of the observation via an autoencoder, in addition to a latent observation map taking latent states + parameters to the latent representation of the observation. This makes the proposed algorithm more principled. Experiments on Kolmogorov flow, Tsunami and Atmospheric modelling show that (i) filtering in latent space outperforms the corresponding method in physical space significantly, (ii) using the proposed approach to handle observations yield stronger results than previous approach considered in LD-EnSF, and (iii) the EnFF filtering backbone outperforms other backbones such as IEnKF.

Critical strengths

(S1) More principled handling of obserations compared to LD-EnSF: Overall, the methodology proposed in the work is similar to LD-EnSF [Xiao et al. 2026]. However, a key difference is how it handles the observation / likelihood during assimilation. The present work addresses a key limitation of Latent EnSF/LD-EnSF, which is that it directly maps the observations to the latent space of state + parameters, making its performance heavily dependent on how well the observations reconstruct the ground truth field (in latent space), in addition to making the handling of observation noise ad-hoc. In contrast, the proposed LD-EnFF framework learns a separate latent space for the observations via a VAE and a forward map taking latent state + parameter to the latent observations, making the inference procedure more principled. This benefit is reflected in the experiments, where using the proposed approach leads to better filtering performance than LD-EnSF-style observation encoding.

(S2) Method operates fully in latent space: The proposed methodology operates entirely on a low-dimensional latent space, with physical-space reconstruction only necessary for evaluation. This enables memory and cost reduction, in addition to MC-based guidance being tractable with small ensemble size.

Critical weaknesses

(W1) Difficult to generalize over observation operators: A key weakness is that the algorithm requires multiple learned components (an LDNet for the latent dynamics, an observation autoencoder, and a latent observation model), which makes it challenging to transfer to different settings without re-training. A critical and commonly occurring example where this might pose an issue is when the observation operator is time-dependent, e.g. when the observation is given by a moving satellite track. It is not clear under the proposed method how one can deal with such settings as it requires amortizing over different observation operators, whose output dimension may also change in time.

In addition, a commonly occurring setting is when the observation operator is nonlinear -- while the proposed methodology handles this setting in theory, the experiments considered all have linear observations. I would have liked to seen at least one experiment where the observation operator is nonlinear to see the extent of the applicability of the method.

(W2) Limited evaluations against other ML-based approaches: In terms of the experiments, the paper only compares against ML methods that are based on EnSF/EnFF, excluding other recently proposed methods such as FlowDAS (cited in the work), or its extension, ForcingDAS [Jia et al. 2026], SDA (cited in the work), and DAISI [Andrae et al. 2026]. Without a comparison, it is difficult to judge the advantage of the proposed approach over these other methods. In addition, I am slightly skeptical of the results from LETKF, the only other baseline that is not based on EnSF/EnFF, which performs very poorly (even worse than pure forecast). The considered setting of linear observations on a regular grid is a setting where LETKF should work quite well, hence I wonder if the authors have tuned the inflation and localization hyperparameters adequately to get this baseline to work (evidently, there is no mention of inflation/localization in the work). Please could the authors confirm this.

[Jia et al. 2026] Jia, Yixuan, et al. "Forcingdas: Unified and robust data assimilation via diffusion forcing." arXiv preprint arXiv:2605.14285 (2026).

[Andrae et al. 2026] Andrae, Martin, et al. "DAISI: Data Assimilation with Inverse Sampling using Stochastic Interpolants." arXiv preprint arXiv:2512.00252 (2025).

Questions for authors
  • In principle, the method should also work without the observation autoencoder by just using Hreal∘Rθ2H_{\text{real}} \circ R_{\theta_2} as the observation operator from the latent dynamics space to physical observations. Is there a reason to consider the observation encoder y↦ϕy \mapsto \phi and learning a separate latent forward model (s,ε)↦y(s, \varepsilon) \mapsto y? Perhaps, an ablation study to compare the results of assimilating observations in latent space vs physical space, will justify the effort of training these extra models. After all, W1 above originates from having to train these extra components, hence I wonder how critical these components are to the method.
  • As mentioned in W2 above, I am not sure why LETKF performs so poorly in the experiments, when the considered setting is not a pathological case where LETKF breaks down.
  • I am unsure about the necessity of the Gaussian approximation for the likelihood considered in Section 3.3. If using MC guidance, there is no requirement for the likelihood to be Gaussian, hence an alternative approach would just be to approximate (11) via Monte Carlo. Have the authors considered this?
  • What are the wall-clock times of the methods? I believe one of the advantages of working in latent space is the improved cost and memory efficiency, hence I am wondering how much speed up we gain by working in latent space, compared to physical space methods (including LETKF, which should be quite fast in the considered setting).
9 comments
AuthorsOpenReview ↗

Response to Reviewer 54Tn (1/5)

We thank the reviewer for the detailed and expert assessment, and for recognizing the more principled handling of observations compared with LD-EnSF and the benefit of operating entirely in the latent space. We address each point below.

W1. Difficult to generalize over observation operators. A key weakness is that the algorithm requires multiple learned components (an LDNet for the latent dynamics, an observation autoencoder, and a latent observation model), which makes it challenging to transfer to different settings without re-training. A critical and commonly occurring example where this might pose an issue is when the observation operator is time-dependent, e.g. when the observation is given by a moving satellite track. It is not clear under the proposed method how one can deal with such settings as it requires amortizing over different observation operators, whose output dimension may also change in time. In addition, a commonly occurring setting is when the observation operator is nonlinear — while the proposed methodology handles this setting in theory, the experiments considered all have linear observations. I would have liked to seen at least one experiment where the observation operator is nonlinear to see the extent of the applicability of the method.

Response. We agree that the learned observation components are tied to the observation operator used for training, and we now state this limitation explicitly in Section 5. Two clarifications may help delimit its scope.

Time-dependent operators. Only the y⋆y^\star operator and the observation encoder depend on the observation operator; LDNet, which learns the latent dynamics and reconstruction, is independent of it. For a new fixed sensor configuration, only these two components need to be retrained. For time-varying operators such as moving satellite tracks, whose output dimension changes over time, both components would need to be conditioned on the sensor configuration, for example by taking sensor coordinates as inputs or using set-based encoders. This is a natural extension that we did not pursue in this work, and we now state this in Section 5.

Nonlinear operators. The formulation does not assume a linear observation operator. HαH_\alpha is trained to regress the noiseless observations Hreal(xj)H_{\mathrm{real}}(\mathbf x_j) from the augmented latent state, whatever the form of HrealH_{\mathrm{real}}, and the encoder is trained on the observations themselves. The only linearization in the likelihood (Eqs. (12)–(15)) concerns the encoder mean with respect to the observation noise, not the observation operator. We agree that all our experiments use linear sub-sampling operators, and an experiment with a nonlinear operator would be a valuable addition; given the time available in the rebuttal period, we prioritized the additional baselines and ablations requested by the reviewers.

AuthorsOpenReview ↗

Response to Reviewer 54Tn (2/5)

W2. Limited evaluations against other ML-based approaches. In terms of the experiments, the paper only compares against ML methods that are based on EnSF/EnFF, excluding other recently proposed methods such as FlowDAS (cited in the work), or its extension, ForcingDAS [Jia et al. 2026], SDA (cited in the work), and DAISI [Andrae et al. 2026]. Without a comparison, it is difficult to judge the advantage of the proposed approach over these other methods. In addition, I am slightly skeptical of the results from LETKF, the only other baseline that is not based on EnSF/EnFF, which performs very poorly (even worse than pure forecast). The considered setting of linear observations on a regular grid is a setting where LETKF should work quite well, hence I wonder if the authors have tuned the inflation and localization hyperparameters adequately to get this baseline to work (evidently, there is no mention of inflation/localization in the work). Please could the authors confirm this.

Response. We added FlowDAS (Chen et al., 2025) as a representative ML-based baseline, as it addresses the same sequential filtering problem: it learns a one-step transition with stochastic interpolants and assimilates the observation at each step. FlowDAS operates in the full physical space; it was trained on the same training trajectories and evaluated on the same observation sequences with 20 samples per step on Kolmogorov flow (due to the limited rebuttal period). It does not estimate the uncertain parameter and does not receive it as input, so we report only the full-state RelRMSE. SDA targets trajectory smoothing, while ForcingDAS and DAISI are very recent works, both published at ICML 2026; we now discuss them in the related work. On Kolmogorov flow, FlowDAS attains a final full-state error of 84.9%, lower than LETKF (122.2%) and LETKF without analysis (121.5%; the same initial ensemble and full numerical model with the analysis step removed), but far above LD-EnFF-OT (5.74%); its error is 5.6% at the first assimilation time and grows over the window.

Regarding LETKF, the reviewer is right that the original manuscript did not report its inflation and localization settings. We now report them in Appendix D.3: multiplicative inflation ρ=1.14\rho=1.14 applied to the prior term of the analysis; hard-cutoff localization with a radius of one observation spacing, so that each local analysis uses the 3×33\times3 neighboring observation locations with all observed channels; and the uncertain parameter included in the augmented state and updated by a global ensemble transform using all observations. The same configuration is used for all three benchmarks. As a no-assimilation reference for LETKF, we added a full-model forecast without assimilation (LETKF without analysis) on Kolmogorov flow: its final error is 121.5±11.7121.5\pm11.7%, essentially the same as that of LETKF (122.2±8.5122.2\pm8.5%). LETKF does not improve on the forecast without assimilation in this setting. We also checked the sensitivity of LETKF on two validation trajectories of Kolmogorov flow, varying the inflation factor ρ∈{1.05,1.14,1.3,1.5}\rho\in\lbrace 1.05,1.14,1.3,1.5\rbrace with a localization radius of one observation spacing, and the radius (one and two observation spacings) at ρ=1.14\rho=1.14. None of these settings consistently reduced both the final and the time-averaged errors relative to the original setting, and larger inflation increased the errors; we therefore kept the original configuration. See also our response to Q2 for a discussion of the potential reasons why LETKF breaks down in this case.

AuthorsOpenReview ↗

Response to Reviewer 54Tn (3/5)

Q1. In principle, the method should also work without the observation autoencoder by just using Hreal∘Rθ2H_{\mathrm{real}}\circ R_{\theta_2} as the observation operator from the latent dynamics space to physical observations. Is there a reason to consider the observation encoder y↦ϕy\mapsto\phi and learning a separate latent forward model (s,ε)↦y(s,\varepsilon)\mapsto y? Perhaps, an ablation study to compare the results of assimilating observations in latent space vs physical space, will justify the effort of training these extra models. After all, W1 above originates from having to train these extra components, hence I wonder how critical these components are to the method.

Response. We thank the reviewer for this question, which goes to the core of the design. We address it from the computational, theoretical, and empirical sides.

Computation. In LDNet, Rθ2R_{\theta_2} decodes the full physical state (dxd_x = 45,000, 67,500, and 393,216 for the three benchmarks). Using Hreal∘Rθ2H_{\mathrm{real}}\circ R_{\theta_2} as the observation operator would require decoding every ensemble member at every cycle; on Kolmogorov flow, this costs 0.045 s per cycle (Appendix D.5), about as much as the entire latent analysis, and the cost grows with dxd_x. The learned y⋆y^\star operator maps the augmented latent state directly to the noiseless sparse observations and avoids this step.

Theory (new Appendix B.4). The observation encoder provides a separate benefit by regularizing the likelihood used in the EnFF analysis. Building on the likelihood-weighted resampling characterization of EnFF by Transue et al. (2025), we analyze how encoding affects the likelihood weights. Under an affine Gaussian approximation with fixed covariance, we prove that the latent likelihood reduces the observation-averaged variance of the log-likelihood weights relative to the direct physical likelihood (Proposition 1). We further establish a corresponding reduction in the expected dispersion of normalized log-weights for any finite ensemble size (Corollary 2). These results provide a theoretical mechanism by which observation encoding can mitigate weight concentration in small ensembles. They do not imply a guaranteed improvement in effective sample size or filtering accuracy.

Ablation (new Appendix E.3). As suggested, we ran LD-EnFF-OT with the observation encoder removed, using the direct likelihood N(yj;Hα(κ),Γy)\mathcal N(\mathbf y_j; H_\alpha(\boldsymbol\kappa),\boldsymbol\Gamma_y) on the physical observations, with everything else unchanged (same LDNet, initial ensembles, observation sequences, NN, KK, and σmin⁡\sigma_{\min}). We use HαH_\alpha rather than Hreal∘Rθ2H_{\mathrm{real}}\circ R_{\theta_2} mainly because it gives a controlled comparison: both likelihoods then share the same forward operator and differ only in whether the misfit is evaluated through the observation encoder.

BenchmarkLikelihoodFinal RelRMSE (%)SSR
Kolmogorov FlowDirect6.70 ± 0.560.205 ± 0.081
Latent (ours)5.74 ± 0.260.536 ± 0.083
Tsunami ModelingDirect5.99 ± 0.550.398 ± 0.124
Latent (ours)6.53 ± 0.250.701 ± 0.122
Atmospheric ModelingDirect4.80 ± 0.430.113 ± 0.017
Latent (ours)4.42 ± 0.210.232 ± 0.025

The table reports mean ±\pm standard deviation over 20 test trajectories (per trajectory, the RelRMSE is averaged and the SSR pooled over three random seeds). The spread–skill ratio (SSR) measures the ensemble spread relative to the estimation error, with values closer to one indicating better calibration and values below one indicating an overconfident ensemble. The latent likelihood improves SSR by a factor of 1.81.8–2.62.6 across all three benchmarks, indicating a less overconfident ensemble. It also yields lower final RelRMSE on Kolmogorov flow and atmospheric modeling, while the direct likelihood performs slightly better on tsunami modeling.

These results show that observation encoding primarily improves the representation of uncertainty rather than uniformly improving the accuracy of the ensemble mean. This is particularly relevant for ensemble filtering, where quantifying uncertainty is an important objective alongside state estimation.

Reviewer_wAdxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this paper, the authors introduced a latent space-based approach dubbed LD-EnFF for data assimilation with limited measurements. The approach combines latent dynamics (via an LDNet module), ensemble flow filtering and a learned latent observation model for forecasting in latent space. The latent space and an uncertain parameter are jointly updated over time with a flow-matching process. The new observations are incorporated through likelihood-based guidance. The method does not require repeated full-state reconstruction. The authors consider multiple tasks like Kolmogorov flow, tsunami modelling, and atmospheric modelling, and show that the proposed LD-EnFF achieves the lowert full-state error among compared latent-space methods.

Critical strengths

The paper offers an interesting approach to perform forecasting in a latent space without full-state reconstruction and leveraging an extra (uncertain) parameter. The writing is clear

Critical weaknesses

While the latent-space approach is interesting, the experimental comparisons appear limited and several aspects require greater explanations and clarity that is missing in the current manuscript. These are listed further in the questions below.

Questions for authors
  1. In Algorithm 1 in Appendix B.3, can the authors detail Step 8 more elaborately? How is EnFF applied to advance the latent state using LjlatL_j^{lat} as observation factor? Please provide the steps for it as well.

  2. Table 1 appears limited as it only includes some latent space methods. More current baselines need to be considered in the data assimilation domain.

  3. What errors are introduced by the method as time/frame increases? A better understanding of the error propagation mechanism is missing although there is some insight e.g., from Figure 2.

  4. In caption of Figure 2, its noted full-space EnFF methods are not shown for Tsunami and atmospheric modelling because the corresponding runs exhibited severe error growth. Could the authors elaborate more on this? Are these the exact equivalent of the proposed methods in full space?

  5. The authors state that for training the y* operator, the training target is the corresponding noiseless sparse physical observation. In practice, real data that could be used to train models won’t be noiseless. What happens here in presence of noise either white or colored noise?

  6. The authors mention the zero-velocity degeneracy issue (e.g., Appendix B.2). Can they clearly specify what the degeneracy is and why is it a major issue?

  7. In Appendix A.3, what does the approximation P≈1P \approx 1 cost in practice? Some more insights are needed. Also, the difference between π\pi and π’\pi’ is not clear.

  8. For equation (31), the authors say “retaining the same conditional probability paths and conditional vector fields, the guidance term is (31)”. Can the authors provide an explanation or reference here?

  9. In Section 3.3, the authors say that the learned y* operator predicts the corresponding noiseless physical observation. It appears the prediction is assumed perfect here?

  10. Is the σθ\sigma_\theta in (13) related to Σenc\Sigma_{enc} in (12)? Is equation (16) an evaluation – i.e., it gives a number? Can equation (18) be explained better?

6 comments
AuthorsOpenReview ↗

Response to Reviewer wAdx (1/4)

We thank the reviewer for the careful reading and the detailed questions, which helped us improve the clarity of the manuscript considerably.

Q1. In Algorithm 1 in Appendix B.3, can the authors detail Step 8 more elaborately? How is EnFF applied to advance the latent state using LjlatL_j^{lat} as observation factor? Please provide the steps for it as well.

Response. Step 8 performs the EnFF analysis in the augmented latent space, and we have added the complete procedure as Algorithm 2 (revised Appendix B.3). Because the guidance depends on the likelihood only through the predictive endpoint, Ljlat\mathcal{L}^{\mathrm{lat}}_j is evaluated once at each forecast member z1(n)=κ^j(n)\boldsymbol z_1^{(n)}=\widehat{\boldsymbol\kappa}_j^{(n)}, giving NN likelihood evaluations per assimilation cycle, independent of the number of flow steps. The particles are then transported from the reference samples (Gaussian samples for OT; perturbed previous posterior members for F2P) by integrating the guided vector field with K=200K=200 explicit Euler steps. For OT, the guided field is the Monte Carlo estimator (33)–(34). For F2P, the guidance term follows the official EnFF code released by Transue et al. (2025), which pairs each particle’s source with all predictive members; it is given in the new Eq. (47) (Appendix B.2). The terminal particles form the posterior ensemble.

Q2. Table 1 appears limited as it only includes some latent space methods. More current baselines need to be considered in the data assimilation domain.

Response. Among recent ML-based DA methods, we added FlowDAS (Chen et al., 2025) as a representative baseline because it addresses the same sequential filtering problem: it learns a one-step transition with stochastic interpolants and assimilates the observation at each step. FlowDAS operates in the full physical space; it was trained on the same training trajectories and evaluated on the same observation sequences with 20 samples per step. It does not estimate the uncertain parameter and does not receive it as input, so we report only the full-state RelRMSE. SDA targets trajectory smoothing, while ForcingDAS and DAISI are very recent works, both published at ICML 2026; we now discuss them in the related work.

Due to the limited rebuttal period, we evaluated FlowDAS on Kolmogorov flow only and report it in a new table in Appendix D.3 (Table 5), together with LETKF and a full-model forecast without assimilation (LETKF without analysis). On Kolmogorov flow, FlowDAS attains a final full-state error of 84.9%, lower than LETKF (122.2%) and LETKF without analysis (121.5%), but far above LD-EnFF-OT (5.74%); its error is 5.6% at the first assimilation time and grows over the window.

AuthorsOpenReview ↗

Response to Reviewer wAdx (2/4)

Q3. What errors are introduced by the method as time/frame increases? A better understanding of the error propagation mechanism is missing although there is some insight e.g., from Figure 2.

Response. To identify the source of the error growth, we added an oracle reference (new Appendix E.2, Figure 4) in which LDNet is run with the true parameter from s0=0\mathbf s_0=\mathbf 0 and without assimilation; its error reflects the accuracy of the learned surrogate when the parameter is known exactly. LD-EnFF-OT closely follows this oracle on all three benchmarks, and LD-EnFF-F2P approaches it once its parameter estimate has converged. The late-stage growth of the full-state error (e.g., on Kolmogorov flow) therefore follows the oracle and stems from the accumulation of surrogate error along the LDNet rollout; this error is not introduced by the analysis step, although the selection-based analysis can reduce it only to the extent that the members differ. In other words, LD-EnFF-OT attains approximately the accuracy of the surrogate run with the true parameter. In contrast, all latent-space methods other than LD-EnFF-OT and LD-EnFF-F2P, which use the same LDNet dynamics, stay above the oracle over most of the window, and on Kolmogorov flow and atmospheric modeling the gap widens in the second half (Figure 4); their final errors on Kolmogorov flow range from 13.09% to 14.40%, versus 5.74% for LD-EnFF-OT (Table 1). Note that “LDNet dynamics only” in Figure 2 uses the initial parameter samples, so its error combines parameter error and surrogate error; the oracle separates the two. In summary (Appendix E.2), over the window, the full-state error of LD-EnFF is dominated early on by the parameter error of the initial ensemble, which decreases as the parameter is identified (Figure 3), and later by surrogate error accumulating along the LDNet rollout. Throughout, the unobserved components, more than 99.5% of the grid, are not observed directly and are determined by the latent state through the learned surrogate. Since the true state at every assimilation time in our benchmarks is determined by the parameter, initial-state uncertainty does not contribute here, and further accuracy gains mainly require improving the fidelity of the surrogate dynamics.

Q4. In caption of Figure 2, its noted full-space EnFF methods are not shown for Tsunami and atmospheric modelling because the corresponding runs exhibited severe error growth. Could the authors elaborate more on this? Are these the exact equivalent of the proposed methods in full space?

Response.

The full-space EnFF methods are the physical-space counterparts of LD-EnFF, rather than exact equivalents. They use the same OT/F2P analysis, Monte Carlo guidance, ensemble size (N=20N=20), and σmin⁡\sigma_{\min}. However, they propagate the augmented physical state using the full numerical model and assimilate observations through the physical likelihood. Their initial ensembles are constructed from the same parameter samples as those of the latent-space methods (Appendix D.4).

The full-space methods exhibited substantial error growth on Tsunami and Atmospheric Modeling. Although we have not isolated the exact cause, possible factors include likelihood weight concentration with a small ensemble, or sensitivity of the full numerical model to parameter mismatch. In particular, when the likelihood concentrates on forecast members that are not close to the truth, errors in the largely unobserved physical state may accumulate over successive assimilation cycles. We have clarified these possible explanations in Appendix D.3.

AuthorsOpenReview ↗

Response to Reviewer wAdx (3/4)

Q5. The authors state that for training the y* operator, the training target is the corresponding noiseless sparse physical observation. In practice, real data that could be used to train models won’t be noiseless. What happens here in presence of noise either white or colored noise?

Response. In model-based DA, training trajectories are generated by numerical simulation, so the noiseless targets Hreal(xj)H_{\mathrm{real}}(\mathbf x_j) are available. The VAE encoder, in contrast, is already trained on noisy observations (Appendix D.2). If only noisy targets yj=yj⋆+ηj\mathbf y_j=\mathbf y_j^\star+\boldsymbol\eta_j were available, with ηj\boldsymbol\eta_j zero-mean and independent of the state, the population minimizer of the squared-error objective (10) would still be the conditional mean E[yj∣κj]=E[yj⋆∣κj]\mathbb E[\mathbf y_j\mid\boldsymbol\kappa_j]=\mathbb E[\mathbf y_j^\star\mid\boldsymbol\kappa_j], i.e., the same target as with noiseless data. This holds for both white and colored noise; the noise only increases the variance of the estimator, which more training data can mitigate. Colored noise is also handled in the likelihood without modification, since Γy\boldsymbol\Gamma_y in (15) may be any symmetric positive-definite covariance, and the term JμΓyJμ⊤J_\mu\boldsymbol\Gamma_yJ_\mu^\top propagates it directly; the encoder itself, including Σenc\boldsymbol\Sigma_{\mathrm{enc}}, would then be trained on observations with the same colored noise. This applies to spatially correlated noise; noise correlated across assimilation times would violate the independence of the observation errors in (2) and would require augmenting the state with the noise process, which we have not considered. See the revised Section 3.2.2 and Appendix D.2.

Q6. The authors mention the zero-velocity degeneracy issue (e.g., Appendix B.2). Can they clearly specify what the degeneracy is and why is it a major issue?

Response. Under the purely memberwise F2P pairing, z0(n)\boldsymbol z_0^{(n)} is a previous posterior member and z1(n)\boldsymbol z_1^{(n)} is its own forecast. Since the forecast leaves ε\boldsymbol\varepsilon unchanged, every conditional velocity z1(n)−z0(n)\boldsymbol z_1^{(n)}-\boldsymbol z_0^{(n)} has a zero parameter component (Eq. (43)). The Monte Carlo guided field is a weighted combination of these velocities, so its parameter component is also zero, whatever the weights. Consequently, ε\boldsymbol\varepsilon would stay at its initial ensemble values regardless of the observations. This is a major issue because joint state–parameter estimation is a central goal of LD-EnFF: an incorrect ε\boldsymbol\varepsilon makes the latent forecasts drift.

In LD-EnFF, the base field keeps the (slightly perturbed) memberwise pairing, whose parameter velocity is only of order σmin⁡\sigma_{\min}, while the guidance term (new Eq. (47)) pairs each particle’s perturbed source with all predictive members. Its parameter component therefore equals ∑n(w~n(ℓ)−wn(ℓ))ε(n)\sum_n(\widetilde w_n^{(\ell)}-w_n^{(\ell)})\boldsymbol\varepsilon^{(n)}, whose scale is set by the ensemble parameter spread; at t=0t=0 it equals the difference between the likelihood-weighted and the uniform ensemble parameter means. This avoids the degeneracy while preserving the dynamical association of each member with its forecast. We have rewritten Appendix B.2 to describe this precisely.

Q7. In Appendix A.3, what does the approximation P≈1P\approx1 cost in practice? Some more insights are needed. Also, the difference between π\pi and π′\pi' is not clear.

Response.

The coupling π\pi connects the reference distribution ρ0\rho_0 to the predictive distribution ρ1\rho_1 and defines the unguided flow, while π′\pi' connects the same reference distribution to the filtering distribution ρ1′∝ρ1e−J\rho_1'\propto\rho_1e^{-J} and defines the guided flow.

For independent couplings, P=1\mathcal P=1 holds exactly. For dependent couplings, setting P=1\mathcal P=1 generally changes the source marginal, introducing bias in the resulting posterior transport. This bias can be significant when the coupling is strongly dependent, as discussed by Feng et al. (2025).

In LD-EnFF-OT, the coupling is independent, so P=1\mathcal P=1 is exact. In LD-EnFF-F2P, P=1\mathcal P=1 is also exact for the guidance evaluated under an independent coupling with each particle’s source fixed. However, the base field uses memberwise dynamical pairing, so combining the two fields does not yield an exact posterior transport. This construction preserves the dynamical pairing while allowing updates of the uncertain parameters. We have clarified these distinctions in the revised Appendices A.3 and B.2.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response (1/2)

We thank all reviewers for their constructive feedback. We have uploaded a revised manuscript in which all changes are highlighted in blue. The main changes are:

  1. Clarified scope. We now state explicitly that LD-EnFF targets joint state–parameter estimation in high-dimensional parameterized dynamical systems, where the state evolution is governed by uncertain parameters. In all three benchmarks, which follow prior latent DA studies (Si & Chen, 2025; Xiao et al., 2026), the simulations start from an initial condition that is either common to all trajectories (Kolmogorov flow, where a burn-in period precedes the assimilation window, and atmospheric modeling) or specified by the uncertain parameter (tsunami modeling), so the true state at every assimilation time is determined by the parameters (Abstract, Sections 1, 4, and 5).

  2. Error growth and an oracle reference (new Appendix E.2). We added an oracle reference in which LDNet is run with the true parameter. LD-EnFF-OT closely follows this oracle on all benchmarks, and LD-EnFF-F2P approaches it once its parameter estimate has converged, whereas all latent-space methods other than LD-EnFF, including LD-EnSF, stay above it over most of the window. We also explain the sources of the full-state error over time.

  3. F2P implementation (Appendix B.2). We expanded the description of the F2P guidance, in which the perturbed point [Eq. (45) in the revised manuscript] serves as the source: the base field uses the perturbed memberwise pairs, while the guidance pairs each particle’s perturbed source with all predictive members [new Eq. (47)], as in the official EnFF code released by Transue et al. (2025). Since the Monte Carlo estimators in the EnFF paper are written with index-matched pairs, we now state this cross-member pairing explicitly. All reported results were obtained with this implementation and are unchanged.

  4. Algorithmic and theoretical clarifications. The full EnFF analysis step is given as Algorithm 2 (Appendix B.3), and the guidance derivation and the role of P\mathcal P are clarified in Appendix A.3.

  5. Theoretical characterization of the analysis step (new Appendix B.4). Building on the characterization of EnFF by Transue et al. (2025, Theorem 4.3 and Remark 4), we note that the OT analysis with Monte Carlo guidance and exact ODE integration is equivalent in distribution to likelihood-weighted resampling of the forecast ensemble followed by Gaussian jitter. This motivates studying how observation encoding affects the likelihood weights. Under an affine Gaussian approximation, we prove that the proposed latent likelihood reduces the observation-averaged variance of the log-likelihood weights relative to the direct physical likelihood (Proposition 1). We further establish a corresponding bound on the expected dispersion of normalized log-weights for any finite ensemble size (Corollary 2), providing a theoretical mechanism for mitigating weight concentration in small ensembles. Finally, we characterize how the time-invariant uncertain parameters evolve under the OT analysis. Although the parameters remain unchanged during forecast propagation, they are updated through likelihood-weighted selection and Gaussian jitter during analysis. We establish a high-probability bound on their displacement from the selected initial ancestors and show that the set of surviving initial ancestors is nonincreasing over assimilation cycles (Corollary 5).

  6. Likelihood ablation (new Appendix E.3). Replacing the latent likelihood by the direct likelihood N(yj;Hα(κ),Γy)\mathcal N(\mathbf y_j;H_\alpha(\boldsymbol\kappa),\boldsymbol\Gamma_y), with everything else unchanged, shows that, with the same forward y⋆y^\star operator, the latent likelihood increases the spread–skill ratio by a factor of 1.8–2.6 and roughly halves the variability of the final error across test trajectories. The final errors of the two likelihoods differ by at most one percentage point (lower with the latent likelihood on Kolmogorov flow and atmospheric modeling, slightly lower with the direct likelihood on tsunami modeling), and both are below those of the LSTM-based baselines on all three benchmarks.

AuthorsOpenReview ↗

General Response (2/2)

  1. Additional baselines. We now report the LETKF settings (Appendix D.3), checked its sensitivity to inflation and localization on Kolmogorov flow, and added a full-model forecast without assimilation as a reference; LETKF does not improve on this forecast on the test trajectories, and none of the tested inflation and localization settings consistently improves on the original configuration. We also report wall-clock times on Kolmogorov flow (Appendix D.5): with all methods on the same GPU, one LD-EnFF-OT cycle is about 23 times faster than LETKF and full-space EnFF-OT. On Kolmogorov flow, the added FlowDAS baseline improves on LETKF but attains a final error of 84.9%, far above that of LD-EnFF-OT (5.74%) (Appendix D.3).

  2. Limitations. We now state that the approximation error of the learned observation operator is not modeled, that uncertainty in the initial state beyond the parameters is not represented by the current LDNet-based formulation, and that the learned observation components are tied to a fixed observation operator (Sections 3.3 and 5, Appendix D.2).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.