acceptodds
Under review as a conference paper at ICLR 2027

Learning Spectral Preconditioner for Linear System Solver with Application to Four-Dimensional Variational Data Assimilation

Abstract

Four-dimensional variational data assimilation (4DVar) requires solving large-scale, ill-conditioned linear systems within its inner loop, where convergence depends on the spectral properties of the system matrix. Traditional limited-memory preconditioners (LMPs) reuse historical Lanczos/Ritz vectors but may lose effectiveness as the relevant subspace rotates between assimilation windows. We propose a neural spectral preconditioner that predicts a current-window subspace from historical features using a Fourier neural operator. The network output is orthonormalized before the preconditioner is constructed. Under the control-variable-transformed model, this constructed preconditioner is symmetric positive definite. Experiments on the strongly nonlinear Lorenz-96 system demonstrate that the proposed method reduces the number of conjugate gradient iterations by 54.16% compared with unpreconditioned CG and by 39.6% compared with LMP. On the quasi-geostrophic (QG) system, the method reduces the iteration count by 22.5% relative to LMP and improves fixed-iteration solution accuracy in the reported tests. The method retains the PCG solution framework and yields consistent reductions in iteration count across the evaluated dynamical regimes.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
AI for science, Four-Dimensional Variational data assimilation, machine learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_BTXhOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this submission, the authors propose a neural network to predict a preconditioner for the 4D-Var data assimilation algorithm. This algorithm relies on Gauss-Newton steps, each requiring a linear solve that is typically ill-conditioned. Existing methods rely on using preconditioning vectors from previous time steps, which is suboptimal because eigenspaces rotate across assimilation windows. The authors propose learning a mapping between the preconditioning vectors of an assimilation window and those of subsequent windows using supervised learning. A spectrally weighted training loss is designed to learn an optimal preconditioning map. Applications to the Lorenz 96 system and a quasigeostrophic model are presented, along with ablation studies. The method considerably improves the condition number of the 4D-Var linear system, thereby enhancing data assimilation accuracy at a fixed computational budget. Results suggest that the method exhibits robustness to changes in underlying dynamics.

Critical strengths

Four-dimensional data assimilation is a key method in operational weather forecasting. Improving its computational efficiency is an important and well-motivated problem.

The proposed method is evaluated on challenging, turbulent physical systems.

The numerical experiments demonstrate significant improvements, bringing performance close to the oracle baseline.

The ablation studies clearly confirm the utility of the proposed spectral loss weighting.

Critical weaknesses

1. Clarity in method exposition

The exposition of the method could be clearer. The paper introduces various matrix notations throughout with sporadic references, making the main objective difficult to follow.

Mathematical quantities are introduced without proper context. For example, Lanczos vectors are never formally defined, and the "invariant subspace matrix" should be precisely defined in terms of its eigendecomposition. Additionally, the exact output dimensions of the network prediction are not clearly specified.

Intermediate linear algebra derivations lack sufficient justification. For instance, the claim that the matrix TGTT_{\mathrm{GT}} is tridiagonal is inadequately explained, and its physicalinterpretation remains vague.

The Limited-Memory Preconditioner (LMP) framework is not properly defined in the text.

2. Computational cost and scalability

A major concern regarding the proposed approach is its computational cost. The method relies on extensive linear algebra operations on high-dimensional objects. In particular, the neural network outputs a matrix whose size scales with the state dimension, raising concerns about practical deployment in large-scale numerical weather forecasting.

The computational complexity of the method should be thoroughly analyzed with respect to state dimension dd and subspace dimension kk, and its applicability to operational weather-scale systems should be critically assessed.

The feasibility and challenges of obtaining ground-truth preconditioning vectors in large-scale systems need to be explicitly discussed.

Questions for authors
  1. Could the authors elaborate on what primarily causes the system matrix AA to be ill-conditioned in this setup?
  2. Could the authors provide a more thorough theoretical and empirical comparison of their method with LMP?
  3. As noted by the authors, the method's computational requirements for large-scale systems depend heavily on implementation choices. Recent works operate variational data assimilation in latent spaces to reduce the scale of optimization variables [1]. Could the authors discuss whether their neural preconditioner can be adapted to or benefit from such latent-space frameworks?

References

[1] Fan, H., Bai, L., Fei, B., Xiao, Y., Chen, K., Liu, Y., ... & Gentine, P. (2026). Physically consistent global atmospheric data assimilation with machine learning in latent space. Science Advances.

2 comments
AuthorsOpenReview ↗

Response to Reviewer BTXh

Thank you for highlighting the spectrally weighted loss and the experiments on the two dynamical systems, and for identifying where the notation, computational cost, and connections to prior work need clarification.

1. Sources of ill-conditioning and notation

The original system matrix AA of 4DVar contains the background precision matrix and a curvature contribution from each observation time. After the control variable transform, the system matrix takes the form S=I+G⊤GS = I + G^{\top}G, where GG propagates background-scaled initial perturbations to observation space and weights them by observation uncertainty. Directions that the observations constrain strongly or that the dynamics amplify can have large curvature. In weakly observed directions, the curvature stays close to the background level. The resulting spread in eigenvalues makes the 4DVar system harder to solve.

Section 3.1 now distinguishes an exact invariant subspace from a truncated Lanczos subspace. For a set of eigenvectors UjU_j of SS, we have SUj=UjΛjSU_j = U_j\Lambda_j. Starting from q1q_1, kk Lanczos steps produce an orthonormal basis VV of span⁡{q1,Sq1,…,Sk−1q1}\operatorname{span}\{q_1,Sq_1,\ldots,S^{k-1}q_1\} before breakdown. The three-term recurrence makes TGT=V⊤SVT_{\mathrm{GT}} = V^{\top}SV tridiagonal in exact arithmetic. For any coefficient vector yy, the quadratic form y⊤TGTyy^{\top}T_{\mathrm{GT}}y gives the curvature along VyVy. Weighting the training loss by TGTT_{\mathrm{GT}} thus gives greater weight to high-curvature directions. Section 3.2.1 also specifies that the network output is reshaped into a d×kd\times k matrix and that QR factorization yields the predicted basis QQ.

2. Theoretical and empirical comparison with LMP

Thank you for asking us to clarify the relationship to LMP. Our LMP baseline reuses the subspace from the previous window. Our method uses the historical basis and its associated analysis increment to predict a subspace for the current window. The purpose of prediction is to align the preconditioning subspace more closely with the current operator. We then solve the linear system with PCG.

Appendix A.1 motivates this distinction by examining how the target subspace evolves between assimilation windows. The principal-angle measurements show that neighboring windows can have similar subspaces overall while the historical subspace is still imperfectly aligned with current spectral directions. This persistence makes the historical basis informative, and the window-to-window variation motivates updating it through prediction. We will make this connection explicit in the main text and refer readers to Appendix A.1.

In the additional comparison, we form T=Q⊤SQT = Q^{\top}SQ using the current operator for each basis and apply the same rule α=ktr⁡(T)\alpha = \frac{k}{\operatorname{tr}(T)}. With the historical and predicted bases, respectively, the mean iteration counts are 9.53 and 7.87 on L96, and 274.70 and 268.48 on QG. Under matched construction rules, prediction therefore reduces mean iterations by 17.4% on L96 and 2.3% on QG. This quantifies the additional benefit of prediction in each system.

3. Computational complexity, training labels, and operational feasibility

For state dimension dd and subspace dimension kk, the network output and the stored basis each require O(dk)O(dk) space. In addition to network inference, setup requires kk applications of the current operator. Thin QR factorization and assembly of Q⊤(SQ)Q^{\top}(SQ) require O(dk2)O(dk^2) arithmetic operations, and factoring the projected matrix costs O(k3)O(k^3). Once these factors are available, each preconditioner application costs O(dk+k2)O(dk+k^2).

We generate training labels offline by running Lanczos on the operator for each training window. Obtaining the kk-step basis and its projection requires approximately kk operator applications per sample and storage for a d×kd\times k basis. During operational use, the network reads historical information to predict the current basis. We have also conducted preliminary tests in an operational assimilation system. See Response 5 to Reviewer mPew.

4. Connection to latent-space assimilation

Thank you for pointing us to Fan et al. (2026), who perform variational assimilation in a learned, low-dimensional representation of the atmospheric state. If the latent inner loop yields a symmetric positive definite system, a subspace predictor could be trained for that operator. For latent dimension mm and fixed kk, the basis would contain mkmk entries instead of dkdk. Section 4.5 cites this work and identifies the combination as a direction for future research.

Thank you for the detailed response. It clarifies the method development and I encourage the authors to include these clarifications in the revised versions of the submission.

The point on computational complexity confirms my concern about larger scale applications. The number of model evaluations for obtaining the ground truth vectors is N×kN \times k, with NN the number of samples, which is at least in the thousands. In operational data assimilation, the dimension dd is at least of order 10710^7, so I imagine kk must be at least a few hundreds. As mentioned in the discussion with reviewer mPew, the system matrices are not directly available, requiring further approximations. Furthermore, linear algebra operations involving the neural network scale with dd, and are thus expensive. These costs look prohibitive for the method in an operationnal setting, and it is not clear if it would scale.

For these reasons, and because of the general lack of clarity in the submission, I maintain my score.

Reviewer_udPiOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a neural spectral preconditioning framework for accelerating the inner-loop optimization of 4DVAR. The model is trained to use historical information to predict a preconditioner, with a rotation-invariant weighted Grassmann metric. Experiments show that the proposed framework performs better than unpreconditioned-CG and LMP under both L96 and QG settings, with good robustness and generalization.

Critical strengths
  1. Well motivated method that is defined rigorously.
  2. The Performance of proposed framework is strong, and shows good robustness and generalization.
Critical weaknesses
  1. Although I understand that this is a proof-of-concept study, the problem itself is from operational systems (as stated by the authors in the Introduction). Therefore, I think the current experiments insufficiently convincing. A more challenging or higher-dimensional experimental setting is still needed, such as a 4DVar data assimilation scenario involving deep learning-based weather forecasting models.
  2. In the fixed-tolerance experiments, only the iteration counts are compared, without any comparison of actual runtime. This does not fully align with the claimed “acceleration”.
Questions for authors

In addition to the issues mentioned in the Weaknesses section, I have the following questions and suggestions:

  1. An ablation study comparing the Grassmann metric with RMSE would make the experiments more comprehensive.
  2. The dataset splitting strategy is not clearly described. Although the paper mentions that temporal correlations have been reduced, the division of the data into training, validation, and test sets should also be explicitly stated.
  3. The choice of FNO seems to be motivated by its sufficiently large global receptive field, but this is only mentioned in the ablation study. A clearer explanation of the motivation before ablation study would help reduce confusion.
  4. The model hyperparameters and training hyperparameters should be clearly specified.
3 comments
AuthorsOpenReview ↗

Response to Reviewer udPi

Thank you for recognizing the motivation for our method and the experiments across assimilation windows and dynamical regimes. We appreciate your suggestions for improving reproducibility.

1. Operational validation and wall-clock time

As a proof-of-concept validation, we evaluate our preconditioner in L96 and two-layer QG experiments, where the state dimensions reach 1,000 and 8,192, respectively. The results reported in the paper demonstrate improved performance.

In our PyTorch implementation, inference, QR factorization, and preconditioner setup offset the time saved during the solve. We have also conducted preliminary tests in an operational weather data assimilation system with state dimension 4×58×320×160=11,878,4004\times58\times320\times160=11{,}878{,}400. Applying our preconditioner reduces the iteration count by 10%. A thorough validation in an operational 4DVar system remains future work.

2. Subspace loss and elementwise error

Thank you for suggesting the comparison. Flipping the sign of a basis vector can change the elementwise error even though it preserves the subspace, its orthogonal projector, and the resulting preconditioner. Table 5 in Section 4.4 now includes a model trained with elementwise mean squared error (MSE). Compared with MSE, the full energy-weighted subspace loss reduces the mean iteration count by 2.16 iterations (21.5%) on L96 and 3.83 iterations (1.4%) on QG. The existing “w/o T-mat” ablation in Table 5 isolates the effect of spectral weighting. The table below reports mean PCG iteration counts.

\textbf{System} & \textbf{Elementwise MSE} & \textbf{Ours} \\\\ \hline \textbf{L96} & 10.03 & 7.87 \\\\ \textbf{QG} & 272.30 & 268.47 \end{array}$$ ## 3. Training, validation, and test data For L96, we use 2,000 of the 2,500 paired samples for training and 500 for validation. For QG, we use 1,600 of the 2,000 pairs for training and 400 for validation. Both splits are random at the level of sample pairs. For each experimental setting, we generate 30 separate test samples to evaluate the neural preconditioner. Section 4.1 now describes both the training/validation split and the generation of the test samples. ## 4. Choice of FNO and implementation details We chose a Fourier neural operator (FNO) so that the prediction at each spatial location can use information from the entire state domain. Section 3.2.1 now explains this motivation when we introduce the Fourier layers. Section 4.4 reports the comparison with a CNN, and Section 4.1 gives the training settings. Regarding implementation details, for both systems, we use AdamW with an initial learning rate of $5\times10^{-4}$ and a weight decay of $10^{-5}$. ReduceLROnPlateau monitors the mean training loss with `mode=min`, `factor=0.5`, and `patience=5`.

Thank you for the detailed response. The additional explanations have helped clarify several aspects of the paper, and I encourage the authors to incorporate these clarifications into the revised manuscript.

I understand the difficulty of conducting additional experiments in operational weather systems within the rebuttal period, and I appreciate the authors' efforts to provide as much evidence as possible under the current constraints.

However, regarding the first response, the authors still do not provide quantitative wall-clock time results, relying instead on reductions in iteration counts. I believe there remains an important distinction between reducing the number of iterations and achieving actual computational acceleration. Furthermore, the authors acknowledge that "In our PyTorch implementation, inference, QR factorization, and preconditioner setup offset the time saved during the solve." This further reinforces my concern, as it suggests that the paper's central claim of acceleration may not be fully supported by the current experimental evidence.

For these reasons, I am unable to increase my score at this stage. Nevertheless, I would still appreciate seeing the actual wall-clock time measurements, including the overhead of preconditioner construction, before finalizing my assessment.

AuthorsOpenReview ↗

Comparison of wall-clock time

To provide a detailed wall-clock time comparison, we report the total runtime, including the costs of network inference, QR factorization, and preconditioner construction.

The comparison is shown in the table below.

SystemCGOurs (precond + solve)Gold (precond + solve)
L96, d=40d=40, k=20k=2017.1720 + 7.87 = 27.8720 + 3.07 = 23.07
QG, d=2048d=2048, k=45k=45461.9045 + 268.48 = 313.4845 + 218.00 = 263.00

As shown, constructing the neural preconditioner incurs additional cost, although our method significantly reduces the solve time. In terms of total runtime on QG, our method outperforms plain CG.

We thank the reviewer for pointing out the distinction between iteration-count reduction and actual wall-clock acceleration. We agree with this distinction, and we have revised the manuscript to frame our contribution as improved convergence efficiency at the iteration level, rather than end-to-end wall-clock speedup.

We thank the reviewer for their time and consideration.

Reviewer_mPewOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper learns a second-level preconditioner for the 4DVar inner loop. An FNO takes the previous window's Lanczos basis and analysis increment and predicts a k-dimensional basis for the current window. It is trained with a subspace projection loss weighted by TGT=VGT⊤SVGTT_{\mathrm{GT}} = V_{\mathrm{GT}}^\top S V_{\mathrm{GT}}. At inference the output is orthonormalized by QR, projected onto the current operator (T=Q⊤SQT = Q^\top S Q), and plugged into a trace-scaled limited-memory preconditioner that is SPD whenever S⪰IS \succeq I. On Lorenz-96 (D up to 1000) and a two-layer QG model (D up to 8192), the method needs fewer PCG iterations than LMP and plain CG, gives lower error at fixed iteration budgets, and stays stable under changes in forcing, window length and sampling gap.

Critical strengths
  1. The solver stays a Krylov method and the preconditioner is SPD whatever the network outputs, so a bad prediction costs speed, not correctness. This is the right way to bring learning into an operational solver.
  2. The iteration savings are consistent across two systems, several dimensions and out-of-distribution settings. LMP degrades with window length while the learned basis does not (L96, window 18: 9.97 vs 20.27 iterations).
Critical weaknesses
  1. The comparison with LMP mixes two changes. Ours and Gold use α=k/tr(T)\alpha = k/\mathrm{tr}(T), LMP uses α=1\alpha = 1. On QG, the method without this scaling needs 695.9 iterations (Table 5), more than LMP (346.5) and even plain CG (461.9). So on QG the gain over LMP may come from the scaling rather than from the learned subspace. The LMP baseline also looks weak as configured: on L96 with D = 1000 it converges more slowly than plain CG (Fig. 1a). LMP with the same projection and trace scaling is the missing control.
  2. Fewer iterations are not shown to mean less compute. Forming TT takes kk applications of SS, plus inference and QR. On L96 with D=40D = 40, k=20k = 20 already exceeds the 17.17 iterations of plain CG (about 28 vs 17 operator applications). There is no wall-clock or operator-count comparison, and no comparison with spending the same kk applications on a current-window Lanczos basis, which is what Gold does.
Questions for authors
  1. How does LMP perform with the same projected TT and α=k/tr(T)\alpha = k/\mathrm{tr}(T) on both systems? How does the method compare with the learned 4DVar preconditioners of Trappler & Vidard (2025) and Moazzami et al. (2025), which the paper cites but does not evaluate?
  2. What are the total operator applications and wall-clock time per solve, including TT, inference and QR? How are the Gold Lanczos vectors generated (starting vector, cost)?
  3. In real cycling, where do the input Lanczos vectors come from? The previous PCG solve produces preconditioned Krylov vectors, not the unpreconditioned Lanczos vectors used in training. How many training samples are used, and how are they split?
  4. The numbers in Sec. 4.3.2 (Ours 6.30 → 9.30, LMP 7.53 → 16.93) do not match Fig. 2(a), which shows roughly 7.0 → 9.4 and 10.6 → 19.4. Which is correct?
3 comments
AuthorsOpenReview ↗

Response to Reviewer mPew(Part 1/2)

Thank you for recognizing the strengths of our solver framework and for your detailed comments on the baseline comparisons.

1. LMP with mismatched projection and scaling

For a fair comparison, we also apply trace scaling to LMP. In this case, LMP with trace scaling differs from our method in the subspace used for preconditioning, while using the same scaling rule, α=ktr⁡(T)\alpha = \frac{k}{\operatorname{tr}(T)}, where T=Q⊤SQT = Q^{\top}SQ and QQ contains the Lanczos vectors from the previous timestep.

The table below reports mean PCG iteration counts.

\textbf{System} & \textbf{LMP + trace scaling} & \textbf{Ours + trace scaling} \\\\ \hline \text{L96} & 9.53 & 7.87 \\\\ \text{QG} & 274.70 & 268.48 \end{array}$$ On both L96 and QG, trace scaling reduces the iteration count relative to LMP without trace scaling. However, it still performs worse than our method. This improvement shows that performance benefits from the corrected subspace. Unlike the existing LMP preconditioner, which is constructed from a subspace shifted by one timestep, our method reduces the mismatch error by using supervised learning to learn the aligned subspace at the **underlying assimilation timestep** in 4DVar. With matched projection and scaling, our method further reduces the mean iteration count by 17.4% on L96 and 2.3% on QG. Both results support the feasibility of our approach. We have added this ablation study to Section 4.4. ## 2. LMP converges more slowly than plain CG For a SPD linear system, the precise convergence rate depends on the operator, the right-hand side, and the initial point. Preconditioning is a common way to improve the worst-case convergence rate. In LMP, the preconditioner is heuristically constructed from the mismatched subspace at the previous timestep. It may therefore adversely affect convergence, causing LMP to converge more slowly than plain CG. Our motivation is to correct this mismatch by predicting the subspace at the aligned timestep. ## 3. Comparison with Trappler and Vidard (2025) and Moazzami et al. (2025) Thank you for suggesting these comparisons. Our paper provides a proof-of-concept validation of a neural preconditioner for the 4DVar problem. In a real operational setting, the assimilation system is high-dimensional, with hundreds of millions of variables, and the system matrix is not explicitly available. Only matrix-vector products can be computed using the tangent linear model and its adjoint. In this setting, the training data include only archived Krylov bases, analysis increments, and low-dimensional projected matrices. These records do not provide access to the original high-dimensional operator $A$ for evaluating $Az$ for an arbitrary vector $z$. Trappler and Vidard (2025) train on random probe vector $z$ and the corresponding products $Az$. Generating these additional products requires operator access beyond what our training interface provides. Moazzami et al. (2025) learn the action of the inverse Hessian from paired right-hand sides and target states and then use the network output to initialize CG. A direct comparison would therefore be unfair to some extent because of the differences in problem settings. We will clarify these differences in the revised manuscript. ## 4. Comparison of wall-clock time To provide a detailed operator applications comparison, we report the total operator applications, including the costs of network inference, QR factorization, and preconditioner construction. The comparison is shown in the table below. $$\begin{array}{c|c|c|c} \textbf{System} & \text{CG} & \textbf{Ours (precond + solve)} & \textbf{Gold (precond + solve)} \\\\ \hline \text{L96},\\ d=40,\\ k=20 & 17.17 & 20 + 7.87 = 27.87 & 20 + 3.07 = 23.07 \\\\ \text{QG},\\ d=2048,\\ k=45 & 461.90 & 45 + 268.48 = 313.48 & 45 + 218.00 = 263.00 \end{array}$$ As shown, constructing the neural preconditioner incurs additional cost, although our method significantly reduces the solve iteration. In terms of total operator applications on QG, our method outperforms plain CG.
AuthorsOpenReview ↗

Response to Reviewer mPew(Part 2/2)

5. Construction of the Gold basis and operational validation

To obtain the Gold basis labels, we run unpreconditioned Lanczos iterations on the current operator SS, starting from a normalized Gaussian random vector. The random seeds are 43 for L96 and 52 for QG, respectively. For the preconditioned linear systems, the initial point is set to zero. Constructing the Gold basis labels and the tridiagonal matrix TT in kk Lanczos steps requires approximately kk matrix-vector products with the current operator.

This proof-of-concept validation provides a foundation for applying our method to a real operational 4DVar system. We have also conducted preliminary tests in an operational weather data assimilation system. The state dimension is 4×58×320×160=11,878,4004\times58\times320\times160=11{,}878{,}400. Applying our preconditioner reduces the iteration count by 10%.

6. Inputs during operational cycling and data splitting

To clarify, our method aims to accelerate forecast prediction and reduce runtime without changing the conventional 4DVar system, thereby ensuring the reliability of the learned model. The original unpreconditioned Lanczos iteration (i.e., CG) is run at every timestep. Hence, the unpreconditioned Lanczos/Ritz vectors used as input to the learned model are always available. For this reason, the paired supervised data consist of Ritz vectors from consecutive assimilation timesteps. We randomly split the paired samples into training and validation sets, using 2,000/500 of the 2,500 L96 pairs and 1,600/400 of the 2,000 QG pairs. We generate 30 separate test samples per setting for evaluation.

7. Discrepancy between Section 4.3.2 and Figure 2(a)

Thank you for catching this discrepancy. Figure 2(a) reports the correct results. We have corrected the values in Section 4.3.2 using the experimental records underlying the figure.

Thank you for adding the matched-scaling comparison and implementation details. The additional iteration reduction on QG is only 2.3%, and the overall computational benefit remains unclear when setup and cycling-input costs are considered. My main concern about practical acceleration is therefore not fully resolved. I will maintain my weak-reject rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.