acceptodds
Under review as a conference paper at ICLR 2027

Multi-Task Learning of Conditional Mean Operators: applications to dynamical systems and uncertainty quantification

Abstract

Estimating conditional statistics and learning representations of a population of conditional distributions are central problems in many data-driven applications, including uncertainty quantification and dynamical systems analysis. Conditional mean operators (CMOs), a class of linear operators between function spaces, resolve these objectives by providing access to a broad class of conditional statistics. However, existing methods typically estimate each CMO independently or constrain it to prespecified function spaces, thereby preventing the exploitation of shared structure across related distributions. In this work, we posit that related CMOs share finite-dimensional input and output function spaces, and are specialized for each task with a linear operator mapping these spaces. Based on this hypothesis, we introduce MTL-CMO, a multi-task framework that jointly learns shared function spaces and task-specific operators across multiple datasets. We further introduce T-CMO, a transfer learning method that reuses the shared spaces to estimate, in closed form, the operator of a new conditional distribution. We establish statistical guarantees quantifying the benefits of jointly learning the shared function spaces. Our experiments demonstrate that learning shared function spaces improves uncertainty quantification across a broad range of conditional distributions and, when applied to Langevin and plasma dynamics, yields compact representations of complex dynamics that retain physically meaningful information and enable parameter identification.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
multi-task learning, operator learning, transfer learning, uncertainty quantification, dynamical systems

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_1UthOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2(−1)
Read review
Paper summary

The paper proposes an extension of the Neural Conditional Probability (NCP) framework of Kostic et al. (2024) for sample-efficiently learning the conditional mean operators (CMOs) of multiple conditional distributions with shared structure. Concretely, the authors adopt a two-sided shared-subspace factorization, assuming that the leading left and right singular functions of all CMOs lie in common low-dimensional function spaces, while each CMO is represented by a task-specific low-rank operator between these spaces. The shared function spaces are learned jointly using a variation of NCP and can be transferred to a new conditional distribution at test time, where the task-specific operator admits a closed-form solution once the shared spaces are frozen. A theoretical bound on the task-averaged excess risk shows increased sample efficiency when the CMOs exhibit substantial shared structure, assuming i.i.d. data and approximate optimality of the learned solution. The benefits of the proposed method are further demonstrated empirically for uncertainty quantification and representation of dynamical systems, with comparisons against various baselines.

Critical strengths
  • Relevant problem and sound contribution. Multi-task learning of CMOs is an interesting and relevant problem with limited prior work. The proposed method is sound, and the capabilities it enables are useful and appear novel. The wide range of applications makes it plausible that the proposed method will spark broad interest and inspire subsequent work.
  • Suitable theoretical and empirical support. While somewhat stylized (assuming i.i.d. data, approximate global optimality of an SGD solution, …), the theoretical result bounding the average excess risk is well-posed to support the advertised sample efficiency. The dependence of the sample efficiency gains on the required dimension of the shared subspaces and their function complexity is natural. The fact that the underlying techniques stay close to classical multi-task learning results, particularly Maurer et al. (2016), is in line with the claimed scope of contributions. The derivations appear sound. Similarly, the empirical experiments illustrate the advantages of multi-task learning CMOs well and situate the proposed method convincingly among several baselines. I do not have the domain expertise in uncertainty quantification or dynamical systems representation to determine whether the scope of the experiments is sufficient to sustain the specific state-of-the-art claim in the introduction, but my overall positive assessment of the paper does not critically depend on this particular claim.
Critical weaknesses
  • Compressed related work. The related work section in the main text is extremely compressed and covers essentially only high-level connections to general multi-task learning results and RKHS theory. While I believe that the paper’s contributions are overall novel for their intended scope, the severely lacking positioning against prior work in the main text currently inflates the perceived novelty scope. In particular, the paper should carefully delineate its scope against [1], which allows multi-task estimation of conditional mean operators with jointly learned input and output spaces parameterized as neural networks, but is missing in the related work. The related work section in the appendix currently also overstates the novelty of two-sided shared feature spaces for multi-task learning more broadly, which has been used before for example in [2].
  • Discussion of trade-offs. The paper could better articulate the assumptions and trade-offs underlying its modeling decisions and results. While shared function spaces are well motivated as a means for efficient multi-task learning, the current presentation provides little intuition for when this assumption is reasonable or restrictive in practice. Similarly, the paper explains why the orthonormality regularization of NCP is dropped, but does not argue why doing so should be inconsequential for the advertised sample-efficiency gains. In principle, removing this regularizer could worsen the conditioning of the learning problem or discard a statistically useful inductive bias, which could offset some of the gains from multi-task learning. While Section 4.1 hints at this briefly and the experiments suggest that this is not a major issue in practice, the question arises naturally and should be addressed explicitly. Finally, the informal statement of Theorem 1 isolates the intended advantage of multi-task learning CMOs well, but neither the surrounding discussion nor the rest of the paper makes sufficiently clear when this advantage actually materializes. As one would expect, there is a trade-off between leveraging samples across multiple tasks and the increased dimension and function-class complexity required to accommodate the shared feature spaces. While this trade-off is entirely natural, it should be discussed explicitly to make clear when multi-task CMO learning is expected to be beneficial.

The above issues are, in my assessment, not major arguments against acceptance and can plausibly be addressed in the revision. The reason I do not lean towards acceptance more strongly is mostly because of the somewhat incremental nature of the proposed method compared to NCP and the fact that the paper currently does not situate the proposed method sufficiently in prior work. Not being an expert in either uncertainty quantification or dynamical systems, the current exposition leaves me uncertain about the exact significance of the contribution. I therefore encourage the authors to explicitly highlight in the rebuttal the significance of their multi-task extension of NCP for their targeted applications.

[1] Ton et al. “Noise contrastive meta-learning for conditional density estimation using kernel mean embeddings” (2021)

[2] Mukherjee et al. “Multi-task Representation Learning for Pure Exploration in Bilinear Bandits” (2023)

Questions for authors
  • Can you explain how your method compares to that of [1] (see weaknesses)? What applications does your approach support that theirs does not?
  • Intuitively, removing the orthonormality regularization from the NCP loss could result in ill-conditioning or remove inductive bias. Have you observed issues with conditioning in your experiments? If so, how did you mitigate it? If not, do you have an intuition for why this was no problem?
  • Your theoretical results (Theorem 1/2) establish sample efficiency for i.i.d. data, but your motivation and experiments emphasize dynamical data. Do your theoretical results extend to more dynamical assumptions on the data-generating process?
  • The gain in sample efficiency advertised by Theorem 1/2 depends on how much larger the required dimension and function complexity terms are for the joint learning problem compared to learning each CMO individually. Can you make this trade-off explicit and explain in what regime the gain in sample efficiency materializes?
  • Can you clarify how the approach scales computationally? In particular, is there a computational advantage of your proposed method over independently applying NCP to each task individually? How well does your method scale (with nn/KK/dd)?
5 comments

W3: Methodological novelty

The above issues are, in my assessment, not major arguments against acceptance and can plausibly be addressed in the revision. The reason I do not lean towards acceptance more strongly is mostly because of the somewhat incremental nature of the proposed method compared to NCP and the fact that the paper currently does not situate the proposed method sufficiently in prior work. Not being an expert in either uncertainty quantification or dynamical systems, the current exposition leaves me uncertain about the exact significance of the contribution. I therefore encourage the authors to explicitly highlight in the rebuttal the significance of their multi-task extension of NCP for their targeted applications.

We thank the reviewer for encouraging us to clarify the significance of our contribution for the targeted applications. Our approach leverages shared structure across related conditional distributions to improve estimation and enable efficient transfer.

For uncertainty quantification, learning shared function spaces improves conditional distribution estimation across tasks. These spaces can then be reused to estimate an unseen distribution through a closed-form matrix fit, without retraining the networks.

For dynamical systems, improved operator estimation enables more accurate recovery of spectral modes, supporting the analysis of dynamical behavior and physical parameter identification. For instance, in the Langevin experiment, with 80k samples, the reconstruction error is 0.075 for T-CMO, compared with 0.115 for MetaKoop, 0.115 for single-task NCP, and 0.088 for VAMPNet. In the plasma experiment, the regressions of parameters κ\kappa and gg have R2 scores of 0.992 and 0.932, respectively.

Our low-rank parametrization allows each task to have distinct dominant singular functions within the shared spaces, whereas directly sharing NCP’s singular functions imposes stronger restrictions. To assess the practical benefit of this flexibility, we added an MTL-NCP baseline in the uncertainty-quantification experiment. This baseline has shared feature maps with task-specific projection matrices and singular values. Note that this baseline forms a novelty in itself since NCP extension to multi-task was not explored. Compare to MTL-NCP, our approach MTL-CMO achieves lower mean Wasserstein-1 error and shorter training times, reducing error by approximately 2–11% and training time by 19–27%. The experiment is detailed in the general response R1.

Beyond this parametrization, we introduce closed-form transfer and establish multi-task and transfer excess-risk guarantees. We further detail the novelty of MTL-CMO in the general response R1.

Q1/W1: Compressed related work.

  • Can you explain how your method compares to that of [1] (see weaknesses)? What applications does your approach support that theirs does not?

We thank the reviewer for the additional references. We were not aware of them; we used them to complete our positioning. We have expanded the related work discussion in the main text and revised the appendix to clarify existing work on two-sided shared representations (see general response R3).

We now discuss our positioning about the mentioned references: [1] MetaCDE. MetaCDE directly estimates conditional density functions through noise-contrastive training and shared feature maps. It uses conditional mean embeddings as intermediate representations. Compared to MTL-CMO, which models conditional distributions via the low-rank representation of the associated CMOs, MetaCDE does not provide access to the CMO's spectral decomposition, which limits its use for analyzing dynamical systems. In addition, as discussed in Section 3.5 of MetaCDE [1], the choice of auxiliary noise distribution affects whether MetaCDE learns the dependence between XX and YY. MTL-CMO simplifies training by estimating the Hilbert–Schmidt loss directly from observed pairs (xi,yi)(x_i,y_i), eliminating the need to specify or sample from an auxiliary noise distribution.

[2] GOBLIN. This multi-task method combines two shared matrix linear subspaces with task-specific inner matrices. They do not investigate learning non-linear embedding with neural netweorks as we do. Furthermore, its objective is to identify the pair of actions with the highest expected reward through active sampling. Our approach differ as it considers shared function subspaces and its objective is to estimate conditional expectation operators from paired observations. Accordingly, its guarantees concern best-action identification, whereas ours concern full operator estimation and transfer.

[1] Ton et al. “Noise contrastive meta-learning for conditional density estimation using kernel mean embeddings” (2021)

[2] Mukherjee et al. “Multi-task Representation Learning for Pure Exploration in Bilinear Bandits” (2023)

Q2/W2: Impact of the absence of orthonormality regularization

Intuitively, removing the orthonormality regularization from the NCP loss could result in ill-conditioning or remove inductive bias. Have you observed issues with conditioning in your experiments? If so, how did you mitigate it? If not, do you have an intuition for why this was no problem?

We did not observe numerical instability attributable to removing the orthonormality regularization during training. Our objective only includes quadratic regularization on the task-specific factors, which discourages excessive factor norms. Our understanding is that training across diverse distributions helps the learning of the shared dictionaries through complementary observations.

To assess the effect of the orthonormality regularization on estimation accuracy, we conducted an additional ablation on the uncertainty-quantification benchmarks. We considered the MTL-NCP model which parametrized each task-specific matrix as Mk=AkDiag(σk)Bk⊤M_k = A_k\text{Diag}(\sigma_k)B_k^\top and applied an orthonormality penalty to the task-specific functions obtained by combining the shared dictionaries with AkA_k and BkB_k. We varied the penalty weight γ\gamma and measured the Wasserstein-1 error. The results show that setting the penalty term is not trivial for MTL-NCP (larger penalties generally worsen it) and support MTL-CMO which does not requires the orthonormality penalty in this setting.

γ\gammaCD1CD2CD3CD4
00.090 ± 0.0020.113 ± 0.0030.116 ± 0.0030.079 ± 0.004
0.0010.090 ± 0.0020.113 ± 0.0030.115 ± 0.0030.080 ± 0.004
0.010.095 ± 0.0020.138 ± 0.0060.115 ± 0.0030.088 ± 0.002
0.10.089 ± 0.0010.173 ± 0.0070.184 ± 0.0120.127 ± 0.002
10.184 ± 0.0070.171 ± 0.0010.629 ± 0.0010.333 ± 0.032
MTL-CMO0.088 ± 0.0040.102 ± 0.0020.106 ± 0.0030.070 ± 0.001

Q3: Extension of theoretical results to dynamical systems

Your theoretical results (Theorem 1/2) establish sample efficiency for i.i.d. data, but your motivation and experiments emphasize dynamical data. Do your theoretical results extend to more dynamical assumptions on the data-generating process?

As the reviewer mentioned, our bounds assume nn i.i.d. pairs per task, but they extend to stationary β\beta-mixing processes through the classical blocking argument (see e.g. Kostic et al [c]). Pairs taken aa time steps apart can be coupled with i.i.d. pairs, up to an additional failure probability controlled by the mixing coefficient β(a)\beta(a). Theorem 2 then holds with nn replaced by an effective sample size neff≍n/an_{\mathrm{eff}}\asymp n/a. For geometrically mixing processes, this costs only a logarithmic factor. The structure of the bound is unchanged: the representation term decreases with KneffKn_{\mathrm{eff}}, while the head term depends on neffn_{\mathrm{eff}} only. The same argument applies to the transfer results. We invite the reviewer to refer to our general response R2, where we provide further details on these extensions.

[c] Kostic, Vladimir, et al. "Learning dynamical systems via Koopman operator regression in reproducing kernel Hilbert spaces." Advances in Neural Information Processing Systems 35 (2022): 4017-4031.

Q4/W2: Efficiency tradeoff

The gain in sample efficiency advertised by Theorem 1/2 depends on how much larger the required dimension and function complexity terms are for the joint learning problem compared to learning each CMO individually. Can you make this trade-off explicit and explain in what regime the gain in sample efficiency materializes?

We thank the reviewer for highlighting this trade-off. As in classical multi-task representation learning, the gain depends on whether the shared representation captures the different tasks without a substantial increase in dimension or complexity.

More explicitly, let Csh=Comp(ΦΘ)+Comp(ΨΘ)\mathcal C_{\mathrm{sh}}=\mathrm{Comp}(\Phi_\Theta)+\mathrm{Comp}(\Psi_\Theta) and Q(d)=ρc2dQ(d)=\rho c^2d. By Theorem 2, the multi-task bound is of order (1+Q(d)) ρcd Csh/Kn+Q(d)(1+Q(d))/n(1+Q(d))\,\rho c\sqrt d\,\mathcal C_{\mathrm{sh}}/\sqrt{Kn}+Q(d)(1+Q(d))/\sqrt n. Applying the same bound to each task separately (K=1K=1), with dictionaries of dimension rr and complexity Cind\mathcal C_{\mathrm{ind}}, gives (1+Q(r)) ρcr Cind/n+Q(r)(1+Q(r))/n(1+Q(r))\,\rho c\sqrt r\,\mathcal C_{\mathrm{ind}}/\sqrt n+Q(r)(1+Q(r))/\sqrt n. Up to constants and logarithmic factors, joint learning is favored when

< (1+Q(r))\big(\rho c\sqrt r\,\mathcal C_{\mathrm{ind}}+Q(r)\big),$$ provided comparable approximation error. Intuitively, a favorable regime consists of related tasks admitting a compact shared representation ($d$ close to $r$) and sufficiently many tasks to offset network complexities. Conversely, representing heterogeneous tasks may require much larger spaces, outweighing the benefit of sharing. Therefore, our bound does not predict an unconditional improvement over task-specific estimation. ## Q5: Scalability > Can you clarify how the approach scales computationally? In particular, is there a computational advantage of your proposed method over independently applying NCP to each task individually? How well does your method scale (with $n/K/d$)? We consider $K$ tasks with $n$ observations each, dictionaries of dimension $d$ and a complexity of the networks' evaluation in $\mathcal{O}(n)$. The computatution costs are: - **Training:** The evaluation of the training objective costs $\mathcal O(Knd^2)$. - **Transfer:** Estimating a new operator costs $\mathcal O(nd^2+d^3)$ for computing empirical moments and performing matrix inversion. It does not require retraining the networks.
Reviewer_9zfZOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes (i) Multi-task Learning of Conditional Mean Operators (MTL-CMO), which learns shared input and output function spaces followed by task-specific low-rank linear operators, and (ii) Transfer-learning CMO (T-CMO), which transfers the learned representations to unseen tasks. The theoretical development of the statistical guarantees is sound, and the experimental part is thorough, clearly showing the strong performance of the proposed methods.

Critical strengths

S1. First, the motivation for jointly estimating CMOs with learned shared spaces is well justified, and it is intuitive why this approach should work. Second, the statistical guarantees are valuable theoretical contributions, even though the analysis is somewhat math-heavy.

Critical weaknesses

W1. My main suggestion is to better position this work relative to previous literature for two reasons: (i) At the architectural level, I think MTL-CMO is quite similar to the well-established shared encoder–latent-decoder operator paradigms in operator learning and multi-task Koopman modeling. (ii) I feel that many key components of this study have been explored in previous literature, including low-rank operator factorization, multi-task Koopman learning, and separable functional representations (e.g., DeepONet and NCP). While I acknowledge the theoretical contributions, I still think the methodological novelty is somewhat limited, and I hope the authors can further clarify what substantial advances MTL-CMO provides beyond these existing approaches.

W2. I think MTL-CMO is closely related to a straightforward multi-task extension of NCP, where neural representations are shared across tasks with task-specific projection matrices and singular values. The main differences seem to lie in regularization and optimization. I wonder if the authors could clarify the methodological advantages of MTL-CMO and consider including such a multi-task NCP baseline for a more convincing comparison.

Questions for authors

Q1. For the abstract, please use one paragraph as required by ICLR format, MTL-CMO and T-CMO are introduced in the abstract but not explained, the author can say “ We introduce Multi-task learning CMO (MTL-CMO) and Transfer-learning CMO (T-CMO)”

Q2. The theoretical analysis assumes independent observations, while the dynamical systems experiments use temporally correlated trajectories. To what extent the statistical guarantees still hold in these settings. Could the authors provide some discussion on this?

2 comments

W1: Positioning with regard to related work

My main suggestion is to better position this work relative to previous literature for two reasons: (i) At the architectural level, I think MTL-CMO is quite similar to the well-established shared encoder–latent-decoder operator paradigms in operator learning and multi-task Koopman modeling. (ii) I feel that many key components of this study have been explored in previous literature, including low-rank operator factorization, multi-task Koopman learning, and separable functional representations (e.g., DeepONet and NCP). While I acknowledge the theoretical contributions, I still think the methodological novelty is somewhat limited, and I hope the authors can further clarify what substantial advances MTL-CMO provides beyond these existing approaches.

We thank the reviewer for encouraging us to clarify how MTL-CMO relates to existing operator learning methods. Drawing on the detailed discussion in Appendix C, we have expanded the related work in the main text. Similar architectures have been explored in multi-task operator learning. Our contribution is to combine this type of architecture with a training objective that directly measures the error of the conditional mean operator. This lets us have a unified framework for uncertainty quantification and dynamical systems analysis, with improved performance in both cases.

Multi-task Koopman learning. The main difference between MTL-CMO and existing Koopman multi-task approaches lies in their training strategies and the implications. As depicted in Appendix C, most existing Koopman methods train their representations by minimizing the state reconstruction error. Accurate state predictions alone do not indicate how accurately the model predicts other quantities. MTL-CMO minimizes a Hilbert–Schmidt discrepancy between operators, which controls their action on square-integrable observables. Our Langevin and plasma experiments illustrate how the resulting operators improved the analysis of dynamical systems.

Operator learning for PDEs. Operator learning approaches to PDEs, such as DeepONet [e] or FNO [f], target a different type of operator compared to MTL-CMO. These approaches learn (time-dependent) solution field maps x↦St(x)x \mapsto S_t(x) of PDEs from initial conditions/external forces given as a function ff, i.e., $

TtfT_t f

(x) = S_t(x)$. These operators are typically learned in a supervised manner from condition/solution pairs, i.e pairs (f,S)(f,S) evaluated at sampled locations. In contrast, MTL-CMO learns a conditional expectation mapping of observables, taking a deterministic dynamical system (Koopman); it corresponds to the operator $

TtfT_t f

(x) = f(S_t(x))$. Considering sampled trajectories, MTL-CMO learns operators in an unsupervised manner thanks to the Hilbert-Schmidt operator loss function. No preset observables are required.

In the general response R3, we discuss and describe the modifications to the main text regarding our positioning with respect to existing approaches.

[e] Lu, Lu, et al. "Learning nonlinear operators via DeepONet based on the universal approximation theorem of operators." Nature machine intelligence 3.3 (2021): 218-229.

[f] Li, Zongyi, et al. "Fourier neural operator for parametric partial differential equations." arXiv preprint arXiv:2010.08895 (2020).

W2: Methodological novelty

I think MTL-CMO is closely related to a straightforward multi-task extension of NCP, where neural representations are shared across tasks with task-specific projection matrices and singular values. The main differences seem to lie in regularization and optimization. I wonder if the authors could clarify the methodological advantages of MTL-CMO and consider including such a multi-task NCP baseline for a more convincing comparison.

We thank the reviewer for encouraging us to better position our work with regard to NCP and its extension to multi-task learning. Following the reviewer’s suggestion, we implemented the proposed multi-task NCP baseline (MTL-NCP) in the uncertainty-quantification experiments. MTL-NCP uses shared feature maps with task-specific projection matrices and singular values. Note that this baseline is novel in itself, since the multi-task setting with NCP has not been investigated previously. As shown below (best performer in bold), MTL-CMO achieves lower mean Wasserstein-1 error and shorter training times across all four benchmarks, reducing error by approximately 2–11% and training time by 19–27%.

CD1CD2CD3CD4
MTL-NCP err. (W1)0.090 ± 0.0020.114 ± 0.0030.119 ± 0.0060.079 ± 0.003
MTL-CMO err. (W1)0.088 ± 0.0040.102 ± 0.0020.106 ± 0.0030.070 ± 0.001
MTL-NCP time (s)34.4 ± 0.923.5 ± 0.424.5 ± 0.119.9 ± 0.0
MTL-CMO time (s)25.6 ± 0.317.2 ± 0.618.3 ± 0.116.1 ± 0.0

Although both parametrizations can represent the same low-rank operators, MTL-CMO avoids enforcing task-specific orthonormality during training. In practice, MTL-NCP was sensitive to the orthonormality penalty weight, which we tuned through a hyperparameter search. These penalties may compete with operator approximation during finite-sample optimization. Our ablation supports this interpretation: small penalties have little effect on Wasserstein-1 error, while larger penalties generally worsen it (see Reviewer 1Uth, Q2/W2). Omitting these penalties also reduces computational overhead per update, providing a plausible explanation for the shorter training times.

Beyond these modeling choices, we introduce closed-form transfer and establish multi-task and transfer excess-risk guarantees, neither of which was provided in the NCP paper. We further detailed thenovelty of MTL-CMO in the general response R1.

Q1: Abstract correction

For the abstract, please use one paragraph as required by ICLR format, MTL-CMO and T-CMO are introduced in the abstract but not explained, the author can say “ We introduce Multi-task learning CMO (MTL-CMO) and Transfer-learning CMO (T-CMO)”

We thank the reviewer for pointing out the incorrect formatting. We directly made the correction in the revised version of the paper.

Q2: Extension of theoretical results to dynamical systems

The theoretical analysis assumes independent observations, while the dynamical systems experiments use temporally correlated trajectories. To what extent the statistical guarantees still hold in these settings. Could the authors provide some discussion on this?

As the reviewer mentioned, our bounds assume nn i.i.d. pairs per task, but they extend to stationary β\beta-mixing processes through the classical blocking argument (see e.g. Kostic et al [c]). Pairs taken aa time steps apart can be coupled with i.i.d. pairs, up to an additional failure probability controlled by the mixing coefficient β(a)\beta(a). Theorem 2 then holds with nn replaced by an effective sample size neff≍n/an_{\mathrm{eff}}\asymp n/a. For geometrically mixing processes, this costs only a logarithmic factor. The structure of the bound is unchanged: the representation term decreases with KneffKn_{\mathrm{eff}}, while the head term depends on neffn_{\mathrm{eff}} only. The same argument applies to the transfer results. We invite the reviewer to refer to our general response R2, where we provide further details on these extensions.

[c] Kostic, Vladimir, et al. "Learning dynamical systems via Koopman operator regression in reproducing kernel Hilbert spaces." Advances in Neural Information Processing Systems 35 (2022): 4017-4031.

Reviewer_jQ52OpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a multi-task learning framework and closed-form transfer learning method for estimating conditional mean operators across populations of related joint distributions. Because each task has its own marginal distributions, the associated Hilbert spaces have task-dependent inner products, making direct SVD parameterisation with task-specific orthogonality constraints cumbersome during joint training. To overcome this, the paper parameterises shared input and output neural dictionaries and unconstrained low-rank task-specific matrices and optimises an unbiased empirical U-statistic estimator. Task-specific orthonormal singular functions are recovered by centring and whitening the learned dictionaries with empirical task-specific Gram matrices. For unseen tasks, authors freeze the learned dictionaries and estimate the target operator in closed form via ridge-regularised Gram matrix inversion. Authors provide statistical learning rates in the IID setting using Gaussian complexity bounds and Hoeffding's U-statistic decomposition, and validate their methods on synthetic conditional CDF estimation, spectral recovery for parametric Müller-Brown Langevin dynamics, and physical parameter identification in 2D turbulent plasma simulations.

Critical strengths
  • Decoupling multi-task representation learning from task-specific orthogonality constraints: In operator learning for conditional distributions, the inner products of the input and output function spaces depend on the marginal distributions of each task. By training shared dictionaries with unconstrained low-rank bilinear heads under unbiased U-statistic loss and leaving task-specific orthonormalisation to a later whitening step the paper provides a practical solution to a central obstacle in multi-task CMO estimation.
  • The combination of T-CMO with the Laplace resolvent generator estimator demonstrates clear utility on challenging physical benchmarks. The inclusion of the Pooled-NCP ablation confirms that learning task-specific operators during training yields substantially better shared fucntion spaces than simply pooling data across tasks
Critical weaknesses

I must preface my comments with the disclaimer that this is a complex topic in which I have very little direct expertise and nearly no experience. I will provide my comments from the point of view of a general statistics/ML researcher and have made an effort to make them as specific as I can to improve the paper. This disclaimer also holds for my questions below.

  • In Sec 3.3 and App G, I believe that the transfer guarantee only bounds ηnew2(θ^)\eta_{\text{new}}^2(\hat{\theta}), which is the population projection error on the new task assuming exact knowledge of the target population Gram and cross-covariance matrices with nnew=∞n_{\text{new}} = \infty. I think that this does not account adequately for the finite-sample estimation error of computing M∗M^* from nnewn_{\text{new}} target observations or the conditioning of empirical Gram matrices G^Φ,G^Ψ\hat G_\Phi, \hat G_\Psi.
  • I also think that Proposition 1(ii) bounds the representation error by the full task risk, but the transfer bound retains a O(d/n)\mathcal{O}(\sqrt{d/n}) term from the source heads that does not vanish as the number of source tasks K→∞K \to \infty for fixed nn.
  • Theorem 1 states a dependence of d\sqrt{d} and dd which I believe hides or at least obscures the fact that in Theorem 2 the Lipschitz factor Lρ=O(d3/2)L_\rho = \mathcal{O}(d^{3/2}) and Bρ=O(d2)B_\rho = \mathcal{O}(d^2) actually yield O(d3/2)\mathcal{O}(d^{3/2}) and O(d2)\mathcal{O}(d^2) dependencies. I am unsure of the practical ramifications of this and would appreciate a clarification here if I have understood it correctly.
  • In Sec 4.1, T-CMO seems to underperform single-task NCP trained from scratch when ntr∈{50,100}n_{\text{tr}} \in \{50, 100\} on CD1 and CD2. But since the dictionary dimension is d=64d = 64, the 64×6464 \times 64 Grams are rank-deficient at ntr=50n_{\text{tr}} = 50 and poorly conditioned at ntr=100n_{\text{tr}} = 100, but in the Appendix D.4 you write the unregularised inverse but don't specify how the regularisation parameter is selected. Does that not cause noise to dominate in the Gram estimation?
  • Additionally, in the multivariate synthetic benchmarks CD2–CD4 all 100 tasks share the exact same 1D projection vector which collapses the 10-dimensional input space to a single shared scalar index and on CD3/CD4 a simple single-task Nadaraya-Watson estimator matches or outperforms MTL-CMO (Table 1).

Nits:

  • Page 3, lines 126–129: The transition operator is denoted AtA_t and then immediately called TtT_t in the next sentence
  • Page 6, after line 323: Typo "sampels" -> "samples".
  • Page 7, line 376 vs Page 29, line 1543: Main text says shared space is learned from 750 systems (with 50 for validation), while Appendix E.2 says 800 training systems (750 + 50 combined).
  • Page 17, line 866: D~Y∣X(k),θ≜PμkDY∣X(k),θPμk\tilde D_{Y|X}^{(k),\theta} \triangleq P_{\mu_k} D_{Y|X}^{(k),\theta} P_{\mu_k} should probably have PνkP_{\nu_k} on the right of the sandwich product?
  • Page 18, lines 952–955: CΦθ,cΨθ,c(k)C_{\Phi_{\theta,c}\Psi_{\theta,c}}^{(k)} is defined with subscript cc but then used without subscript cc in 955.
  • Page 25, lines 1343–1348 & Figure 7 on page 22: CD1–CD4 are referred to as CP1–CP4, I think they should be CD
Questions for authors
  • Gram regularisation in T-CMO. How was ε\varepsilon chosen in the transfer experiments of Figure 2 and Table 5? When ntr=50<d=64n_{\text{tr}} = 50 < d = 64, does tuning ε\varepsilon (or applying shrinkage) help or even eliminate the performance degradation of T-CMO in the low-data regime?
  • In CD2–CD4, w\mathbf{w} is identical across all tasks, meaning the shared input dictionary only needs to learn functions of a single 1D projection w⊤x\mathbf{w}^\top \mathbf{x}. How does MTL-CMO perform if wk\mathbf{w}_k varies across tasks within a pp-dimensional subspace (e.g., p=3p = 3 or 55), so that tasks share a multi-dimensional input function space rather than a single linear index?

Suggestions for future work:

  • Extending Proposition 2 and Theorem 2 to include the finite-sample estimation error of the ridge-regularised target estimator M^∗\widehat{M}^* from finite nnewn_{\text{new}} target observations would make the theoretical transfer guarantee fully reflective of the T-CMO algorithm. I'm also interested if a finite-sample formal bound exists.
4 comments

W1: Clarification on excess-risk bounds

  • In Sec 3.3 and App G, I believe that the transfer guarantee only bounds ηnew2(θ^)\eta^2_{new}(\hat\theta), which is the population projection error on the new task assuming exact knowledge of the target population Gram and cross-covariance matrices with nnew→∞n_{new} \to \infty. I think that this does not account adequately for the finite-sample estimation error of computing from target observations or the conditioning of empirical Gram matrices.
  • I also think that Proposition 1(ii) bounds the representation error by the full task risk, but the transfer bound retains a O(d/n)\mathcal{O}(\sqrt{d/n}) term from the source heads that does not vanish as the number of source tasks K→∞K \to \infty for fixed nn .
  • Theorem 1 states a dependence of d\sqrt{d} and dd which I believe hides or at least obscures the fact that in Theorem 2 the Lipschitz factor Lρ=O(d3/2)L_\rho = \mathcal{O}(d^{3/2}) and Bρ=O(d2)B_\rho = \mathcal{O}(d^{2}) actually yield O(d3/2)\mathcal{O}(d^{3/2}) and O(d2)\mathcal{O}(d^{2}) dependencies. I am unsure of the practical ramifications of this and would appreciate a clarification here if I have understood it correctly.
  • Extending Proposition 2 and Theorem 2 to include the finite-sample estimation error of the ridge-regularised target estimator M^\widehat{M} from finite nnewn_{new} target observations would make the theoretical transfer guarantee fully reflective of the T-CMO algorithm. I'm also interested if a finite-sample formal bound exists.

We thank the reviewer for their careful reading of our paper. We agree with the points raised, which call for more precision and for treating a natural question we had not initially addressed. We are grateful for these suggestions and have completed the analysis accordingly.

1. Finite-sample error of T-CMO. We agree. Our transfer guarantee concerns ηnew2(θ^)\eta^2_{\mathrm{new}}(\hat\theta), the population error of T-CMO, as stated after Propositions 1 and 2. The estimation of the new head from nnewn_{\mathrm{new}} samples, including the conditioning of the empirical Gram matrices, was not treated. We have added this analysis for the ridge estimator used by T-CMO (Proposition 3, Corollary 1 and Lemma 1, Appendix G.3). The estimation error depends on the effective dimension of the ridge-regularized dictionaries instead of the ambient dimension dd, up to an additional bias term. The practical choice of ε\varepsilon is discussed in our answer to Q1.

2. Source-head term. Indeed, and this is consistent with our result: the bound separates the cost of the shared representation, paid with the nKnK samples of all tasks, from the cost of the heads, each estimated from the nn samples of its own task. As stated after Theorem 1 (Section 3.3), this task-specific term depends on nn alone, as is standard in multi-task representation learning (e.g., Tripuraneni et al., 2020 [d]). We thank the reviewer for highlighting this point, and we have made the transfer statement in Appendix G.4 more explicit: "the representation part of the transfer error decreases with nKnK, while the head term depends on nn only."

3. Dependence on dd. We agree that the informal Theorem 1 should make this dependence explicit. The formal Theorem 2 contains it through Qρ=ρc2dQ_\rho=\rho c^2d, and the revised Theorem 1 (Section 3.3) now states the dependence on QρQ_\rho. Appendix G.4 discusses its origin: QρQ_\rho is a worst-case bound over the hypothesis class, and we expect it to be conservative, since trained kernels approximate pk−1p_k-1, which does not depend on dd.

4. Finite-sample formal bound. We thank the reviewer for this suggestion. We have added an analysis of the T-CMO ridge estimator to complete our statistical analysis (point 1, Appendix G.3).

[d] Tripuraneni, N., Jordan, M. I., & Jin, C. "On the theory of transfer learning: The importance of task diversity." NeurIPS 2020.

Q1: Impact of the regularization term in transfer learning setting

  • In Sec 4.1, T-CMO seems to underperform single-task NCP trained from scratch when ntr∈50,100n_{tr} \in 50, 100 on CD1 and CD2. But since the dictionary dimension is d=64d =64, the $64 \times 64 Grams are rank-deficient at ntr=50n_{tr}=50 and poorly conditioned at ntr=100n_{tr}=100, but in the Appendix D.4 you write the unregularised inverse but don't specify how the regularisation parameter is selected. Does that not cause noise to dominate in the Gram estimation?
  • Gram regularisation in T-CMO. How was ε\varepsilon chosen in the transfer experiments of Figure 2 and Table 5? When ntr=50<d=64n_{tr}=50 < d=64, does tuning ε\varepsilon (or applying shrinkage) help or even eliminate the performance degradation of T-CMO in the low-data regime?

We thank the reviewer for pointing out this omission. In the original experiments, ε\varepsilon was selected per MTL dataset on a validation set considering the full trajectory (ntr=400n_{\mathrm{tr}}=400). We clarified the procedure and introduce the regularization explicitly in Appendix D.4. As the reviewer mentionned, at ntr=50n_{\mathrm{tr}}=50, the 64×6464\times64 Gram matrices are rank-deficient, making regularization important to limit noise amplification.

We conducted an additional sensitivity analysis which varies ε\varepsilon at ntr=50n_{\mathrm{tr}}=50 on 10 seeds. Results are displayed below and compared to NCP, the bold entries correspond to the lowest Wasserstein-1 error achieved for each distribution.

CD / ε\varepsilon10−610^{-6}10−410^{-4}0.010.010.10.111NCP
CD10.374 ± 0.0120.312 ± 0.0110.251 ± 0.0120.209 ± 0.0080.237 ± 0.0050.2362 ± 0.0059
CD20.564 ± 0.0160.557 ± 0.0160.400 ± 0.0130.280 ± 0.0050.217 ± 0.0030.2326 ± 0.0082
CD30.610 ± 0.0300.379 ± 0.0210.245 ± 0.0080.489 ± 0.0170.748 ± 0.0040.2603 ± 0.0062
CD40.340 ± 0.0130.306 ± 0.0090.217 ± 0.0040.165 ± 0.0030.313 ± 0.0110.2584 ± 0.0038

A better regularization accounting for the sample size improves the error, and even removes the observed gap in mean performance relative to NCP. However, excessive regularization can worsen accuracy. These results highlight the importance of correctly selecting ε\varepsilon in the low-data regime which was not done in our original experiments where ϵ\epsilon was not adapted to ntrn_{tr}.

Q2: Uncertainty quantification under task-specific projection

  • Additionally, in the multivariate synthetic benchmarks CD2–CD4 all 100 tasks share the exact same 1D projection vector which collapses the 10-dimensional input space to a single shared scalar index and on CD3/CD4 a simple single-task Nadaraya-Watson estimator matches or outperforms MTL-CMO (Table 1).
  • In CD2–CD4, w\mathbf{w} is identical across all tasks, meaning the shared input dictionary only needs to learn functions of a single 1D projection w⊤x\mathbf{w^\top x}. How does MTL-CMO perform if wkw_k varies across tasks within a -dimensional subspace (e.g., p=3p=3 or 55), so that tasks share a multi-dimensional input function space rather than a single linear index?

We thank the reviewer for suggesting this experiment. We extended CD2–CD4 to task-specific projection vectors wkw_k varying within a shared subspace of dimension q∈{1,2,3,5}q\in\{1,2,3,5\}.

We evaluate this setting on the transfer experiment: the learned dictionaries are frozen, and T-CMO estimates each new operator with ntr=400n_{\mathrm{tr}}=400 observations. The table reports Wasserstein-1 error against single-task NCP trained from scratch with best performer in bold for each distribution.

Projection settingCD2 T-CMOCD2 NCPCD3 T-CMOCD3 NCPCD4 T-CMOCD4 NCP
q=1q=10.092 ± 0.0210.143 ± 0.0320.099 ± 0.0290.176 ± 0.0440.063 ± 0.0120.127 ± 0.025
q=2q=20.115 ± 0.0280.143 ± 0.0330.143 ± 0.0390.178 ± 0.0430.069 ± 0.0140.131 ± 0.026
q=3q=30.101 ± 0.0240.143 ± 0.0320.109 ± 0.0260.176 ± 0.0430.068 ± 0.0300.128 ± 0.022
q=5q=50.118 ± 0.0340.142 ± 0.0310.111 ± 0.0410.176 ± 0.0420.067 ± 0.0150.128 ± 0.026

T-CMO retains lower mean error across all tested settings. In particular, for q=3q=3 and q=5q=5, the reductions relative to NCP are approximately 17–30% on CD2, 37–38% on CD3, and 47% on CD4. These results show that the transfer benefit persists when tasks depend on different directions within a multidimensional subspace. Results for sample sizes ranging from ntr=50n_{tr}=50 to 400400 will be included in the Appendix.

Thank you for your extensive response. My concerns are addressed and I am happy to increase my score.

We thank the reviewer for their comments that improved out paper, as well as for their acknowledgment of our rebuttal and raised score.

Comments on the paper

2 comments

We thank all reviewers for their thoughtful reviews, which helped us improve our work. In what follows, we address remarks that appear across several reviews.
Our rebuttal also includes a revised version of the paper with an updated related work section. Changes are colored in blue. Additional results presented in the rebuttal will also be included in the paper.

R1: Novelty of MTL-CMO and T-CMO with regard to NCP [a] (R.9zfZ & R.1Uth).

Our method builds on NCP’s operator-estimation principle. Our contributions concern learning shared function spaces across a population of CMOs, transferring these spaces to unseen distributions, and establishing statistical guarantees for both settings.

1. From a single-operator SVD to shared function spaces with task-specific low-rank operators. NCP directly parametrizes an operator via its SVD: the networks represent singular functions, and a diagonal matrix encodes the singular values. MTL-CMO instead learns shared dictionaries whose spans approximate the dominant singular functions across tasks. Each task has its own low-rank matrix within these spaces, allowing distinct singular functions and effective ranks. Since orthogonality depends on task-specific marginals, directly sharing NCP’s singular functions would impose different orthogonality requirements on the same networks. Our factorization avoids these constraints during training while retaining access to task-specific SVDs via post hoc whitening (Section 3.1). We assess the practical benefit of this flexibility on the uncertainty-quantification experiment by including a new baseline: MTL-NCP. This baseline is the extension of NCP to the multi-task setting; it combines shared feature maps with task-specific projectors and singular values, i.e., Mk=AkDiag(σk)Bk⊤M_k = A_k \text{Diag}(\sigma_k)B_k^\top. As in NCP, the training loss includes task-specific orthonormality penalties. In practice, the penalty weights are difficult to set, so we tuned them via hyperparameter tuning. Note that this MTL-NCP is novel in itself, since the multi-task setting with NCP has not been investigated previously. Results are displayed in the table below, the best performers are in bold. Across all four distribution families, MTL-CMO achieves lower mean Wasserstein-1 error and shorter training times, with error reductions of approximately 2–11% and time reductions of 19–27%.

CD1CD2CD3CD4
MTL-NCP err. (W1)0.090 ± 0.0020.114 ± 0.0030.119 ± 0.0060.079 ± 0.003
MTL-CMO err. (W1)0.088 ± 0.0040.102 ± 0.0020.106 ± 0.0030.070 ± 0.001
MTL-NCP time (s)34.4 ± 0.923.5 ± 0.424.5 ± 0.119.9 ± 0.0
MTL-CMO time (s)25.6 ± 0.317.2 ± 0.618.3 ± 0.116.1 ± 0.0

2. Transfer through closed-form operator estimation. Unlike NCP, which only addresses the estimation of a single conditional distribution, we also propose a transfer procedure (T-CMO, Section 3.2) which reuses the learned function spaces to estimate an unseen conditional distribution by estimating only the task-specific matrix in closed form, without retraining the networks.

3. Statistical guarantees for multi-task and transfer learning: In our work, the multi-task statistical guarantee extend classical guarantees [b] to operator estimation. The bound separates the cost of learning shared dictionaries, which decreases with the total source sample size KnKn, from task-specific estimation, which depends on nn. We also provide novel transfer guarantees accounting for the learned dictionaries (Section 3.3 and Appendix G). These results address the statistical benefit of sharing across distributions, in contrast to NCP which only provides guarantees for estimating a single distribution.

R2: Extension of statistical guarantees to dynamical systems (R.9zfZ & R.1Uth)

Our excess-risk bounds assume nn i.i.d. pairs per task. For dynamical systems, they extend to stationary β\beta-mixing processes through the classical blocking argument as in the paper [c]. For task kk, observed along a trajectory with lag τ\tau, keep the neff=⌊n/a⌋n_{\mathrm{eff}}=\lfloor n/a\rfloor pairs (Xt,Xt+τ)(X_t,X_{t+\tau}) whose starting times are a>τa>\tau apart. By the coupling lemma for β\beta-mixing sequences, these pairs can be replaced by i.i.d. pairs from ρk\rho_k, except on an event of probability at most neff βk(a−τ)n_{\mathrm{eff}}\,\beta_k(a-\tau). Theorem 2 then holds with nn replaced by neffn_{\mathrm{eff}}, with probability at least 1−δ−neff∑k=1Kβk(a−τ).1-\delta-n_{\mathrm{eff}}\sum_{k=1}^K\beta_k(a-\tau). The gap aa trades the effective sample size for the mixing term. For geometrically mixing processes, βk(h)≤Cβe−γh\beta_k(h)\le C_\beta e^{-\gamma h}, choosing a−τ≍γ−1log⁡(Kn/δ)a-\tau\asymp\gamma^{-1}\log(Kn/\delta) gives neff≍n/(τ+γ−1log⁡(Kn/δ))n_{\mathrm{eff}}\asymp n/(\tau+\gamma^{-1}\log(Kn/\delta)), i.e. a loss of only a logarithmic factor. The structure of the bound is unchanged: the representation term still decreases with KneffKn_{\mathrm{eff}}, while the head term depends on neffn_{\mathrm{eff}} only. The same argument applies to the target trajectory in Proposition 3 and Corollary 1. We added this discussion in Appendix G.

R3: A more detailed related work (R.9zfZ & R.1Uth).

To benefit from the additional page, we dedicated a section to related work (see Section 3.4 of the revised paper), expanded the detailed related work in Appendix C, included references suggested by reviewers, and improved the positioning of our work with respect to existing approaches.

Essentially, MTL-CMO is a fine combination of low-rank operator modeling and a theoretically grounded training strategy that enables efficient and accurate estimation of multiple operators, supported by statistical guarantees for training and transfer.

To support this claim, we re-organized the related work in three paragraphs:

  • Multi-task representation learning. Approaches supported by statistical guarantees primarily focus on task-specific predictors built on a shared input feature map. The case of shared input and output feature maps has been studied in the particular context of shared linear subspaces. MTL-CMO extends this two-sided formulation to conditional mean operators by learning shared input/output spaces of nonlinear functions.

  • Conditional mean operator and distribution. Most approaches focus on estimating a single operator or a distribution. MTL-CMO builds on one of the learning approaches, NCP [a]. It extends the NCP's learning objective to multiple tasks and employs low-rank modeling with two shared dictionaries that span function spaces, thereby removing the orthonormality regularization required by the NCP's modeling. This difference in modeling has been shown to improve performance, as depicted in our general response R1.

  • Multi-task operator learning for PDEs and dynamical systems. Operator learning for PDEs targets a different type of operator than MTL-CMO. They learn solution fields of PDEs from initial conditions/external forces given as functions. We made that distinction clear in the revised version of the paper. Closer to MTL-CMO are multi-task Koopman methods. These methods are typically trained to reconstruct trajectories in the state spaces using decoders. In contrast, MTL-CMO training directly targets the discrepancy between operators and provides control over the estimation of the conditional expectation of square-integrable observables.


[a] Kostic, Vladimir R., et al. "Neural conditional probability for uncertainty quantification." Advances in Neural Information Processing Systems 37 (2024): 60999-61039.

[b] Maurer, Andreas, Massimiliano Pontil, and Bernardino Romera-Paredes. "The benefit of multitask representation learning." Journal of Machine Learning Research 17.81 (2016): 1-32.

[c] Kostic, V., et al. "Learning dynamical systems via Koopman operator regression in reproducing kernel Hilbert spaces." NeurIPS 2022.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.