- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes FERA, a data assimilation method that learns a forecast-conditioned error representation to recover forecast error from partial information. It uses held-out-variable and sparse observation objectives for training, then combines the learned representation with historical and ensemble statistics in a linear-Gaussian Bayesian update. FERA outperforms the strongest matched baseline in 46 of 48 ensemble-size diagnostic comparisons in a month-long global cycling experiment. The benefits of forecast conditioning and partial information training are supported by ablation studies.
Critical strengths
The main strength is that FERA connects representation learning to recovery from partial information, which the representation must support during assimilation. The retraining ablations show that the held-out-variable and sparse-observation objectives improve cycling DA.
Critical weaknesses
The global cycling experiments give useful evidence for FERA’s error representation, but all use noise-free observations. Handling observation noise is a basic requirement in DA, and the baselines (VAE-Var and LoRA-EnVar) have already been tested with real observations. A controlled cycling comparison with noisy observations is the main missing experiment.
Questions for authors
-
How did you choose the checkpoints and observation-error scalings for FERA and the baselines in the main comparison? In particular, what validation data, selection criteria, and tuning budgets did you use? These details would help readers interpret the reported results and guide users in tuning FERA.
-
I would be inclined to change my recommendation if there is a controlled cycling comparison with noisy observations.
5 comments
Rebuttal from Authors (2/3)
Q2 and the weakness. Controlled cycling with noisy and incomplete observations
We have completed 29-day cycling experiments (116 six-hour cycles, 3 January–1 February 2024) comparing FERA, 3DVar, and VAE-Var. Each cycle samples 1,000 horizontal columns on the 721×1440 grid and observes all 69 channels there, following the synthetic sparse-column observation convention used in prior weather-DA work (Huang et al., 2024; Xiao et al., 2025). The methods share the forecast model, initial background, observation locations, and noise realizations, then follow their own cycling trajectories.
At each observed location, noise is generated as
Here is the forecast-model normalization standard deviation and is the fixed channel error coefficient. Gaussian draws are independent across observed locations and channels and refreshed each cycle; methods receive identical draws for each condition. We report κ={0, 0.1, 0.25, 0.5}. For scale, κ=0.25 corresponds to noise standard deviations of approximately 25.15 m²/s² for z500, 0.129 g/kg for q850, 0.500 m/s for u500, and 0.164 K for t500.
The injected noise and the solver's R are separate quantities: κ scales noise standard deviation, while scales the solver variance. We test α={1, 10, 30, 100, 300, 1000} for each baseline, retaining 3DVar's 10 outer iterations/LBFGS maxiter=5 and VAE-Var's 10/maxiter=10. FERA retains its direct posterior solve and calibrated R. Thus, α=1 refers to the newly calibrated R described under Q1.
The table reports each baseline's post-hoc best joint scaling on the monthly grid, with one α shared across all diagnostics. Improvements are , using all twelve diagnostics and all 116 cycles.
| Observation noise κ | Selected 3DVar α | Selected VAE-Var α | FERA improvement over 3DVar | FERA improvement over VAE-Var |
|---|---|---|---|---|
| 0 | 1 | 1 | 3.77% | 3.06% |
| 0.1 | 10 | 10 | 5.44% | 4.40% |
| 0.25 | 100 | 30 | 26.13% | 20.12% |
| 0.5 | 300 | 100 | 44.68% | 40.39% |
FERA's advantage becomes larger as observation noise increases within the reported range, even after selecting the best tested scaling for each baseline. Across the three reported nonzero noise levels and all six baseline scalings, FERA achieves lower time-mean WRMSE on every diagnostic (432/432 noise × scaling × baseline × diagnostic comparisons).
The variable-level results show that the gains extend across geopotential, humidity, wind, and temperature. Below are absolute analysis WRMSEs averaged over the same 116 cycles; lower is better. The selected α is shared by all variables, as specified above.
| Variable (unit) | 3DVar κ=0 | VAE-Var κ=0 | FERA κ=0 | 3DVar κ=0.25 | VAE-Var κ=0.25 | FERA κ=0.25 |
|---|---|---|---|---|---|---|
| z500 (m²/s²) | 48.87 | 47.00 | 44.21 | 88.52 | 80.91 | 45.09 |
| z850 (m²/s²) | 39.33 | 38.06 | 38.79 | 68.00 | 57.86 | 39.61 |
| q500 (g/kg) | 0.4730 | 0.4687 | 0.4668 | 0.5265 | 0.4912 | 0.4697 |
| q850 (g/kg) | 1.1943 | 1.1777 | 1.1322 | 1.3840 | 1.2568 | 1.1380 |
| u500 (m/s) | 2.119 | 2.080 | 2.018 | 2.602 | 2.381 | 2.043 |
| u850 (m/s) | 1.691 | 1.672 | 1.655 | 2.008 | 1.847 | 1.672 |
| v500 (m/s) | 2.082 | 2.034 | 2.009 | 2.521 | 2.335 | 2.037 |
| v850 (m/s) | 1.678 | 1.659 | 1.637 | 2.004 | 1.851 | 1.653 |
| t500 (K) | 0.572 | 0.555 | 0.554 | 0.771 | 0.702 | 0.563 |
| t850 (K) | 0.851 | 0.853 | 0.845 | 1.048 | 0.988 | 0.854 |
For example, FERA reduces z500 WRMSE by 49.07%/44.28% relative to 3DVar/VAE-Var at κ=0.25. At this noise level, FERA achieves lower WRMSE on all ten pressure-level diagnostics, including humidity, wind, and temperature.
Rebuttal from Authors (3/3)
Cross-season and observation-seed validation. We additionally evaluated 72 seven-day trajectories across four 2023 seasonal windows (January, April, July, and October), three observation seeds, κ=0/0.25, and all three methods. These windows lie outside the 2024 NMC training year. Checkpoints and R remain fixed across seasons and seeds, and the methods share paired initial backgrounds and observations. The table averages paired relative improvements across the ten pressure-level diagnostics and all twelve season–seed conditions:
| Observation noise κ | FERA improvement over 3DVar | FERA improvement over VAE-Var |
|---|---|---|
| 0 | 7.06% | 4.75% |
| 0.25 | 15.87% | 11.37% |
FERA retains a positive mean advantage in each of the four seasonal windows at both noise levels. At κ=0.25, all 120/120 season × seed × variable comparisons favor FERA against each baseline.
To also represent incomplete variable coverage, we ran seven-day experiments (28 cycles) at κ=0 and 0.25, withholding all 13 humidity levels or all wind observations, including 10-m winds. We retain the full-observation R settings and iteration budgets, without retuning for the masks. The following table reports FERA's mean relative WRMSE reduction on the withheld variables at 500/850 hPa:
| Withheld observations | κ | Improvement over 3DVar | Improvement over VAE-Var |
|---|---|---|---|
| All humidity | 0 | 8.41% | 28.46% |
| All humidity | 0.25 | 9.68% | 10.10% |
| All wind | 0 | 22.70% | 23.65% |
| All wind | 0.25 | 28.74% | 27.48% |
The individual withheld-variable results are shown below. Humidity rows use the all-humidity mask; wind rows use the all-wind mask. Values are analysis WRMSE averaged over the complete 28 cycles; lower is better.
| Withheld variable (unit) | κ | 3DVar analysis | VAE-Var analysis | FERA analysis |
|---|---|---|---|---|
| q500 (g/kg) | 0 | 0.5622 | 0.7146 | 0.5364 |
| q500 (g/kg) | 0.25 | 0.5896 | 0.6003 | 0.5403 |
| q850 (g/kg) | 0 | 1.4165 | 1.8278 | 1.2434 |
| q850 (g/kg) | 0.25 | 1.4050 | 1.3924 | 1.2503 |
| u500 (m/s) | 0 | 4.0576 | 4.1954 | 3.0240 |
| u500 (m/s) | 0.25 | 4.4364 | 4.4477 | 3.0441 |
| u850 (m/s) | 0 | 2.9098 | 2.9105 | 2.5068 |
| u850 (m/s) | 0.25 | 3.1970 | 3.1047 | 2.5316 |
| v500 (m/s) | 0 | 3.9736 | 4.0743 | 2.8478 |
| v500 (m/s) | 0.25 | 4.2823 | 4.2904 | 2.8635 |
| v850 (m/s) | 0 | 2.9092 | 2.8933 | 2.2361 |
| v850 (m/s) | 0.25 | 3.2050 | 3.0748 | 2.2548 |
FERA achieves lower mean analysis WRMSE than both baselines for all six withheld diagnostics at both noise levels. Its advantage therefore extends to variables without direct observations, supporting its effectiveness under incomplete variable coverage.
These results support FERA's effectiveness under both noisy observations and missing-variable coverage.
VAE-Var (Xiao et al., 2025) has also been evaluated with real observations. However, we do not have a matched set of processed observations and preprocessing configurations for that benchmark, which prevents a directly controlled comparison with its reported results. We will investigate real-observation assimilation in future work.
References
- Huang, L., Gianinazzi, L., Yu, Y., Dueben, P. D., and Hoefler, T. (2024). DiffDA: a Diffusion model for weather-scale Data Assimilation. Proceedings of the 41st International Conference on Machine Learning (ICML), Proceedings of Machine Learning Research, 235, 19798–19815.
- Xiao, Y., Jia, Q., Chen, K., Bai, L., and Xue, W. (2025). VAE-Var: Variational Autoencoder-Enhanced Variational Methods for Data Assimilation in Meteorology. International Conference on Learning Representations (ICLR).
Rebuttal from Authors (1/3)
Thank you for recognizing the value of learning forecast-conditioned error representations that recover errors from partial information. We address your questions and the concern about noise-free observations below using completed observation-error calibration, controlled noisy-observation cycling, and additional seasonal and variable-withholding experiments. FERA retains its advantage after baseline R tuning, with larger gains as observation noise increases over the reported range.
Q1. Checkpoint and observation-error scaling selection
The checkpoints used in the original comparison were selected empirically during method development, rather than through a matched checkpoint-search protocol. In particular, the VAE-Var baseline was retrained following the architecture, hyperparameters, and training settings of the original VAE-Var paper (Xiao et al., 2025).
To make error-scaling selection more systematic and comparable, we applied a common calibration protocol for all three methods. We set R=sRpaper, preserving each method's channel ratios, and test the same six variance multipliers on 20–24 December 2023. Each candidate runs for 16 six-hour cycles, with cycles 5–16 used for selection. The score is Qm(s)=121∑cWRMSEm,1,cWRMSEm,s,c,
where the twelve diagnostics are z/q/u/v/t at 500/850 hPa, mean sea-level pressure, and 2-m temperature. Lower scores are better; scores are normalized within each method, so they measure scaling sensitivity rather than cross-method accuracy.
The selected scalings are 0.0625, 0.25, and 0.5 for 3DVar, VAE-Var, and FERA, respectively. Relative to using the original R in this calibration, the normalized score improves by 0.363%, 0.523%, and 0.062%. Thus, all three methods benefit modestly, and FERA's original R already performs very close to its best tested clean-observation setting. This is consistent with learned correction structure and direction being more important for stable assimilation than fine control of the overall observation-error magnitude within the tested range. The noisy experiments below separately test whether its advantage survives a broader baseline R search.