- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2(−1)
Read review
Paper summary
The paper proposes an extension of the Neural Conditional Probability (NCP) framework of Kostic et al. (2024) for sample-efficiently learning the conditional mean operators (CMOs) of multiple conditional distributions with shared structure. Concretely, the authors adopt a two-sided shared-subspace factorization, assuming that the leading left and right singular functions of all CMOs lie in common low-dimensional function spaces, while each CMO is represented by a task-specific low-rank operator between these spaces. The shared function spaces are learned jointly using a variation of NCP and can be transferred to a new conditional distribution at test time, where the task-specific operator admits a closed-form solution once the shared spaces are frozen. A theoretical bound on the task-averaged excess risk shows increased sample efficiency when the CMOs exhibit substantial shared structure, assuming i.i.d. data and approximate optimality of the learned solution. The benefits of the proposed method are further demonstrated empirically for uncertainty quantification and representation of dynamical systems, with comparisons against various baselines.
Critical strengths
- Relevant problem and sound contribution. Multi-task learning of CMOs is an interesting and relevant problem with limited prior work. The proposed method is sound, and the capabilities it enables are useful and appear novel. The wide range of applications makes it plausible that the proposed method will spark broad interest and inspire subsequent work.
- Suitable theoretical and empirical support. While somewhat stylized (assuming i.i.d. data, approximate global optimality of an SGD solution, …), the theoretical result bounding the average excess risk is well-posed to support the advertised sample efficiency. The dependence of the sample efficiency gains on the required dimension of the shared subspaces and their function complexity is natural. The fact that the underlying techniques stay close to classical multi-task learning results, particularly Maurer et al. (2016), is in line with the claimed scope of contributions. The derivations appear sound. Similarly, the empirical experiments illustrate the advantages of multi-task learning CMOs well and situate the proposed method convincingly among several baselines. I do not have the domain expertise in uncertainty quantification or dynamical systems representation to determine whether the scope of the experiments is sufficient to sustain the specific state-of-the-art claim in the introduction, but my overall positive assessment of the paper does not critically depend on this particular claim.
Critical weaknesses
- Compressed related work. The related work section in the main text is extremely compressed and covers essentially only high-level connections to general multi-task learning results and RKHS theory. While I believe that the paper’s contributions are overall novel for their intended scope, the severely lacking positioning against prior work in the main text currently inflates the perceived novelty scope. In particular, the paper should carefully delineate its scope against [1], which allows multi-task estimation of conditional mean operators with jointly learned input and output spaces parameterized as neural networks, but is missing in the related work. The related work section in the appendix currently also overstates the novelty of two-sided shared feature spaces for multi-task learning more broadly, which has been used before for example in [2].
- Discussion of trade-offs. The paper could better articulate the assumptions and trade-offs underlying its modeling decisions and results. While shared function spaces are well motivated as a means for efficient multi-task learning, the current presentation provides little intuition for when this assumption is reasonable or restrictive in practice. Similarly, the paper explains why the orthonormality regularization of NCP is dropped, but does not argue why doing so should be inconsequential for the advertised sample-efficiency gains. In principle, removing this regularizer could worsen the conditioning of the learning problem or discard a statistically useful inductive bias, which could offset some of the gains from multi-task learning. While Section 4.1 hints at this briefly and the experiments suggest that this is not a major issue in practice, the question arises naturally and should be addressed explicitly. Finally, the informal statement of Theorem 1 isolates the intended advantage of multi-task learning CMOs well, but neither the surrounding discussion nor the rest of the paper makes sufficiently clear when this advantage actually materializes. As one would expect, there is a trade-off between leveraging samples across multiple tasks and the increased dimension and function-class complexity required to accommodate the shared feature spaces. While this trade-off is entirely natural, it should be discussed explicitly to make clear when multi-task CMO learning is expected to be beneficial.
The above issues are, in my assessment, not major arguments against acceptance and can plausibly be addressed in the revision. The reason I do not lean towards acceptance more strongly is mostly because of the somewhat incremental nature of the proposed method compared to NCP and the fact that the paper currently does not situate the proposed method sufficiently in prior work. Not being an expert in either uncertainty quantification or dynamical systems, the current exposition leaves me uncertain about the exact significance of the contribution. I therefore encourage the authors to explicitly highlight in the rebuttal the significance of their multi-task extension of NCP for their targeted applications.
[1] Ton et al. “Noise contrastive meta-learning for conditional density estimation using kernel mean embeddings” (2021)
[2] Mukherjee et al. “Multi-task Representation Learning for Pure Exploration in Bilinear Bandits” (2023)
Questions for authors
- Can you explain how your method compares to that of [1] (see weaknesses)? What applications does your approach support that theirs does not?
- Intuitively, removing the orthonormality regularization from the NCP loss could result in ill-conditioning or remove inductive bias. Have you observed issues with conditioning in your experiments? If so, how did you mitigate it? If not, do you have an intuition for why this was no problem?
- Your theoretical results (Theorem 1/2) establish sample efficiency for i.i.d. data, but your motivation and experiments emphasize dynamical data. Do your theoretical results extend to more dynamical assumptions on the data-generating process?
- The gain in sample efficiency advertised by Theorem 1/2 depends on how much larger the required dimension and function complexity terms are for the joint learning problem compared to learning each CMO individually. Can you make this trade-off explicit and explain in what regime the gain in sample efficiency materializes?
- Can you clarify how the approach scales computationally? In particular, is there a computational advantage of your proposed method over independently applying NCP to each task individually? How well does your method scale (with //)?
5 comments
Q2/W2: Impact of the absence of orthonormality regularization
Intuitively, removing the orthonormality regularization from the NCP loss could result in ill-conditioning or remove inductive bias. Have you observed issues with conditioning in your experiments? If so, how did you mitigate it? If not, do you have an intuition for why this was no problem?
We did not observe numerical instability attributable to removing the orthonormality regularization during training. Our objective only includes quadratic regularization on the task-specific factors, which discourages excessive factor norms. Our understanding is that training across diverse distributions helps the learning of the shared dictionaries through complementary observations.
To assess the effect of the orthonormality regularization on estimation accuracy, we conducted an additional ablation on the uncertainty-quantification benchmarks. We considered the MTL-NCP model which parametrized each task-specific matrix as and applied an orthonormality penalty to the task-specific functions obtained by combining the shared dictionaries with and . We varied the penalty weight and measured the Wasserstein-1 error. The results show that setting the penalty term is not trivial for MTL-NCP (larger penalties generally worsen it) and support MTL-CMO which does not requires the orthonormality penalty in this setting.
| CD1 | CD2 | CD3 | CD4 | |
|---|---|---|---|---|
| 0 | 0.090 ± 0.002 | 0.113 ± 0.003 | 0.116 ± 0.003 | 0.079 ± 0.004 |
| 0.001 | 0.090 ± 0.002 | 0.113 ± 0.003 | 0.115 ± 0.003 | 0.080 ± 0.004 |
| 0.01 | 0.095 ± 0.002 | 0.138 ± 0.006 | 0.115 ± 0.003 | 0.088 ± 0.002 |
| 0.1 | 0.089 ± 0.001 | 0.173 ± 0.007 | 0.184 ± 0.012 | 0.127 ± 0.002 |
| 1 | 0.184 ± 0.007 | 0.171 ± 0.001 | 0.629 ± 0.001 | 0.333 ± 0.032 |
| MTL-CMO | 0.088 ± 0.004 | 0.102 ± 0.002 | 0.106 ± 0.003 | 0.070 ± 0.001 |
Q3: Extension of theoretical results to dynamical systems
Your theoretical results (Theorem 1/2) establish sample efficiency for i.i.d. data, but your motivation and experiments emphasize dynamical data. Do your theoretical results extend to more dynamical assumptions on the data-generating process?
As the reviewer mentioned, our bounds assume i.i.d. pairs per task, but they extend to stationary -mixing processes through the classical blocking argument (see e.g. Kostic et al [c]). Pairs taken time steps apart can be coupled with i.i.d. pairs, up to an additional failure probability controlled by the mixing coefficient . Theorem 2 then holds with replaced by an effective sample size . For geometrically mixing processes, this costs only a logarithmic factor. The structure of the bound is unchanged: the representation term decreases with , while the head term depends on only. The same argument applies to the transfer results. We invite the reviewer to refer to our general response R2, where we provide further details on these extensions.
[c] Kostic, Vladimir, et al. "Learning dynamical systems via Koopman operator regression in reproducing kernel Hilbert spaces." Advances in Neural Information Processing Systems 35 (2022): 4017-4031.
Q4/W2: Efficiency tradeoff
The gain in sample efficiency advertised by Theorem 1/2 depends on how much larger the required dimension and function complexity terms are for the joint learning problem compared to learning each CMO individually. Can you make this trade-off explicit and explain in what regime the gain in sample efficiency materializes?
We thank the reviewer for highlighting this trade-off. As in classical multi-task representation learning, the gain depends on whether the shared representation captures the different tasks without a substantial increase in dimension or complexity.
More explicitly, let and . By Theorem 2, the multi-task bound is of order . Applying the same bound to each task separately (), with dictionaries of dimension and complexity , gives . Up to constants and logarithmic factors, joint learning is favored when
< (1+Q(r))\big(\rho c\sqrt r\,\mathcal C_{\mathrm{ind}}+Q(r)\big),$$ provided comparable approximation error. Intuitively, a favorable regime consists of related tasks admitting a compact shared representation ($d$ close to $r$) and sufficiently many tasks to offset network complexities. Conversely, representing heterogeneous tasks may require much larger spaces, outweighing the benefit of sharing. Therefore, our bound does not predict an unconditional improvement over task-specific estimation. ## Q5: Scalability > Can you clarify how the approach scales computationally? In particular, is there a computational advantage of your proposed method over independently applying NCP to each task individually? How well does your method scale (with $n/K/d$)? We consider $K$ tasks with $n$ observations each, dictionaries of dimension $d$ and a complexity of the networks' evaluation in $\mathcal{O}(n)$. The computatution costs are: - **Training:** The evaluation of the training objective costs $\mathcal O(Knd^2)$. - **Transfer:** Estimating a new operator costs $\mathcal O(nd^2+d^3)$ for computing empirical moments and performing matrix inversion. It does not require retraining the networks.
W3: Methodological novelty
We thank the reviewer for encouraging us to clarify the significance of our contribution for the targeted applications. Our approach leverages shared structure across related conditional distributions to improve estimation and enable efficient transfer.
For uncertainty quantification, learning shared function spaces improves conditional distribution estimation across tasks. These spaces can then be reused to estimate an unseen distribution through a closed-form matrix fit, without retraining the networks.
For dynamical systems, improved operator estimation enables more accurate recovery of spectral modes, supporting the analysis of dynamical behavior and physical parameter identification. For instance, in the Langevin experiment, with 80k samples, the reconstruction error is 0.075 for T-CMO, compared with 0.115 for MetaKoop, 0.115 for single-task NCP, and 0.088 for VAMPNet. In the plasma experiment, the regressions of parameters κ and g have R2 scores of 0.992 and 0.932, respectively.
Our low-rank parametrization allows each task to have distinct dominant singular functions within the shared spaces, whereas directly sharing NCP’s singular functions imposes stronger restrictions. To assess the practical benefit of this flexibility, we added an MTL-NCP baseline in the uncertainty-quantification experiment. This baseline has shared feature maps with task-specific projection matrices and singular values. Note that this baseline forms a novelty in itself since NCP extension to multi-task was not explored. Compare to MTL-NCP, our approach MTL-CMO achieves lower mean Wasserstein-1 error and shorter training times, reducing error by approximately 2–11% and training time by 19–27%. The experiment is detailed in the general response R1.
Beyond this parametrization, we introduce closed-form transfer and establish multi-task and transfer excess-risk guarantees. We further detail the novelty of MTL-CMO in the general response R1.
Q1/W1: Compressed related work.
We thank the reviewer for the additional references. We were not aware of them; we used them to complete our positioning. We have expanded the related work discussion in the main text and revised the appendix to clarify existing work on two-sided shared representations (see general response R3).
We now discuss our positioning about the mentioned references: [1] MetaCDE. MetaCDE directly estimates conditional density functions through noise-contrastive training and shared feature maps. It uses conditional mean embeddings as intermediate representations. Compared to MTL-CMO, which models conditional distributions via the low-rank representation of the associated CMOs, MetaCDE does not provide access to the CMO's spectral decomposition, which limits its use for analyzing dynamical systems. In addition, as discussed in Section 3.5 of MetaCDE [1], the choice of auxiliary noise distribution affects whether MetaCDE learns the dependence between X and Y. MTL-CMO simplifies training by estimating the Hilbert–Schmidt loss directly from observed pairs (xi,yi), eliminating the need to specify or sample from an auxiliary noise distribution.
[2] GOBLIN. This multi-task method combines two shared matrix linear subspaces with task-specific inner matrices. They do not investigate learning non-linear embedding with neural netweorks as we do. Furthermore, its objective is to identify the pair of actions with the highest expected reward through active sampling. Our approach differ as it considers shared function subspaces and its objective is to estimate conditional expectation operators from paired observations. Accordingly, its guarantees concern best-action identification, whereas ours concern full operator estimation and transfer.
[1] Ton et al. “Noise contrastive meta-learning for conditional density estimation using kernel mean embeddings” (2021)
[2] Mukherjee et al. “Multi-task Representation Learning for Pure Exploration in Bilinear Bandits” (2023)