acceptodds
Under review as a conference paper at ICLR 2027

GGMPs: Generalized Gaussian Mixture Processes for Repeated-Measure Conditional Density Estimation

Abstract

Gaussian processes (GPs) offer a principled nonparametric framework for learning conditional densities with calibrated uncertainty, but their Gaussian predictive form cannot represent multimodal or strongly non-Gaussian outputs. We introduce the Generalized Gaussian Mixture Process (GGMP), a GP-based method for learning multimodal conditional densities from repeated-measure or distribution-valued data, where each input carries many output samples or an empirical conditional density. Exact inference in a fully coupled mixture-of-GPs model is exponentially intractable; the GGMP instead fits local Gaussian mixtures, aligns their components across inputs, and trains one heteroscedastic GP per component, yielding a closed-form Gaussian mixture predictive density that remains compatible with standard and scalable GP solvers. We prove that under component separation, smooth component trajectories, accurate local fits, and dense inputs, the alignment stage recovers the true component correspondence, and we show when component collisions make that correspondence unrecoverable for any method. On synthetic and real-world repeated-measure datasets the GGMP is competitive with neural and random-forest conditional density estimators, and controlled ablations map its operating range.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
GGMPs learn multimodal conditional densities from repeated-measure data by fitting local Gaussian mixtures, provably aligning components across inputs, and smoothing each with a heteroscedastic GP, giving a closed-form mixture predictive.

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_MzG8OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes GGMP, an approach to overcome the computational burden of inference on mixtures of GPs, caused by the marginalization over assignments. The method introduces three-stage pipeline that avoids exponential complexity and works with scalable GP primitives.

The main contribution is the proposed three-stage pipeline and corresponding theoretical results. The authors also conducted rigorous experiments on various datasets and ablations.

Critical strengths
  1. The paper presents a methodology that can be useful in a field of Gaussian Processes as it provides a scalable training procedure for GP Mixture models.
  2. The proposed method is supported by the corresponding propositions and theorems.
  3. The authors conducted rigorous experiments on various datasets and on various metrics and important ablation studies.
Critical weaknesses
  1. The importance of GP Mixture models is not well-motivated in the Introduction and in general Introduction does not highlight the limitation of other probabilistic approaches in comparison to GP Mixtures.
  2. It is not clear from Experiments section if the goal of the paper is to provide a new probabilistic approach based on GP Mixtures that outperforms other probabilistic methods on specific metrics or the goal is the local improvement on GP Mixtures. If it is the first one, then the experiments do not fully justify it as for many examples in the paper MDN performs on the same level or even better. If it is the second one, it would be good also to compare with some other mixture method utilizing GPs, for example Nguyen, T. & Bonilla, E. (2014). “Fast Allocation of Gaussian Process Experts.” ICML 2014. Also it would be good to provide stds for CRPS metric as well.
Questions for authors

In section 3.3.1, πn(k)\pi_n(k) appears without being introduced before, can authors explain it as it causes troubles with understanding this section.

If authors address Questions and Weaknesses, I would be willing to increase my score.

3 comments
AuthorsOpenReview ↗

Response to Reviewer MzG8

We sincerely thank the reviewer for the positive assessment of the paper, particularly the methodology, theoretical contributions, and experimental studies. We respond to the concerns below:

Motivation for GP mixture models: We agree that the Introduction could better motivate the use of GP mixtures over other probabilistic approaches. We will revise it to emphasize their advantages in repeated-measure settings, where GP priors provide smooth interpolation and uncertainty quantification across inputs, while mixtures allow us to model multimodal conditional distributions. We will also better highlight the limitations of existing CDE approaches in this setting and the computational challenges of conventional GP mixtures, motivating GGMP as an alternative.

Empirical performance and intended contribution: Our goal is the former of the two: to introduce a new probabilistic approach based on GP mixtures, rather than merely improve existing GP mixture methods. However, our objective is not to universally outperform MDNs or other conditional density estimators across every metric. Rather, we emphasize that GGMP provides a tractable and interpretable framework specifically for repeated-measure conditional density estimation, with theoretical guarantees. Our experiments demonstrate that GGMP is competitive with existing CDE methods while offering substantial computational advantages in the tested settings. We believe that the framework and the results together establish GGMP as a strong alternative in its intended setting, and will clarify this positioning in the manuscript.

CRPS standard deviations: We agree that including standard deviations for CRPS would make the comparisons more complete, and will add these to the revised tables.

Notation in Section 3.3.1: We apologize for the confusion regarding pi_n(k), which denotes the locally fitted component index corresponding to the globally aligned component k at input x_n. We will revise the prose to introduce this notation more clearly.

We thank the reviewer again for their constructive feedback. We hope our responses address their concerns and, as indicated in the review, would appreciate reconsideration of the rating.

Thank you for your response! Although I really appreciate the response, the authors have not provided stds for CRPS and have not commented on my propose to compare with some other mixture method utilizing GPs, and I have not found any modifications in the pdf file, so currently I would keep my score.

We thank the reviewer for following up, and apologize that the revised manuscript had not yet been uploaded when they checked. The revised PDF is now available, and we address the remaining concerns below.

CRPS standard deviations: We have now added CRPS standard deviations to the experimental tables. The principal comparisons at K=25 are:

DatasetGGMP-25MDN-25RFCDE
Synthetic0.4846 ± 0.04010.4934 ± 0.08010.4924 ± 0.1054
Temperature4.7765 ± 1.20215.2364 ± 1.49336.0085 ± 1.8229
Manufacturing0.1904 ± 0.03280.1663 ± 0.07420.1951 ± 0.0565

GGMP-25 attains the lowest mean CRPS on the synthetic and temperature benchmarks, while MDN-25 achieves a lower mean CRPS on manufacturing. We have revised the experimental discussion to emphasize these differences in mean performance rather than implying universal superiority.

Comparison with GP-based mixture methods: Our goal is to introduce a new GP-mixture-based probabilistic approach for repeated-measure conditional density estimation, rather than a local improvement to existing GP-mixture methods. We have added Nguyen and Bonilla (2014) to Section 2. Their method focuses on scalable regression through the allocation of pointwise observations among GP experts. In contrast, GGMP uses repeated observations to estimate local mixture distributions and aligns their components across inputs before GP regression. The methods therefore address different modeling settings. Our experiments compare against MDNs and RFCDE as general-purpose conditional density estimators applicable to the same repeated-measure task. We do not claim that GGMP universally outperforms these alternatives; but instead, the results demonstrate its complementary predictive and computational strengths in its intended setting.

We thank the reviewer again for the constructive feedback. We hope the revised manuscript and these clarifications address the outstanding concerns and would appreciate reconsideration of the rating.

Reviewer_FkPfOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper worked on conditional density estimation when multioutput are corresponding to the training input. The proposed model GGMP fits a Gaussian mixture to the output at each input where the means of each mixture component are model with GP. The component predictions and learned mixture weights yield an analytic Gaussian-mixture density at a new input. Experiements cover synthetic data and real world datasets.

Critical strengths

GGMP reduces many observations at each input to local mixture parameters, then fits GPs across the N distinct inputs. Conditional on those fitted parameters, GP inference and the final mixture density are analytic. This is a useful computational pattern when the number of repeats is large relative to N.

Critical weaknesses

The experiments do not establish a broad accuracy or efficiency advantage yet. MDNs have lower synthetic symmetric KL and L1L^1 error at matched component counts. GGMP has better CRPS and coverage on the temperature task, but it draws on summaries of roughly 50 million observations, while MDN and RFCDE train on 10 million pairs. That study also uses single split. The manufacturing result comes from 24 conditions, with five held out per split. Its runtime comparison uses different hardware for GGMP and MDN. Since dense GGMP training also scales as O(KN3)O(KN^3) it remains unclear whether these gains hold with matched data exposure or many more distinct inputs..

Questions for authors

Can the temperature comparison use matched data exposure? For example, GGMP could use a station-stratified 10 million observation sample, or the MDN could train on all available observations. This would help separate the effect of the model from the effect of using more data.

3 comments
AuthorsOpenReview ↗

Response to Reviewer FkPf

We sincerely thank the reviewer for the thoughtful assessment of our paper and for recognizing the computational advantages of GGMP. We respond to the concerns below:

Empirical performance and intended contribution: We would like to clarify that our objective is not to universally outperform MDNs or other conditional density estimators across every metric. Rather, GGMP provides a tractable GP-based framework specifically for repeated-measure conditional density estimation, with interpretable component trajectories and theoretical guarantees on alignment, identifiability, and consistency. Our experiments demonstrate that GGMP is competitive with existing CDE methods, outperforming them on certain metrics while offering substantial computational advantages in the tested settings. Indeed, while MDNs achieve lower symmetric KL and L1 errors on the synthetic benchmark, GGMP attains a lower mean CRPS and coverage closer to nominal at higher component counts, illustrating the complementary strengths of the approaches. We believe these complementary results establish GGMP as a strong alternative in its intended setting, and have clarified this positioning in the revised manuscript.

Matched data exposure for the climate dataset: We thank the reviewer for this suggestion. We have now rerun GGMP on the same station-stratified 10M observation subset matching the MDN and RFCDE baselines across three seeds. The results are nearly identical to those obtained using the full dataset:

ModelSym. KLL1CRPSCov. 90%
GGMP-3 (10M)2.5632 ± 1.53230.5101 ± 0.24027.2492 ± 1.91120.8906
GGMP-10 (10M)2.6002 ± 1.54330.4922 ± 0.25125.8533 ± 1.54020.9034
GGMP-25 (10M)2.5653 ± 1.63420.4903 ± 0.24444.7692 ± 1.20180.8989

GGMP-25's CRPS changes only from 4.7765 to 4.7692, and its coverage from 0.8987 to 0.8989. GGMP-25 still attains a lower mean CRPS than MDN-25 (4.7692 ± 1.2018 vs. 5.2364 ± 1.4933) and coverage closer to the nominal 90% (0.8989 vs. 0.8438), while achieving comparable symmetric KL and L1 errors. These results show that GGMP's lower mean CRPS and better coverage calibration persist under matched data exposure, instead of being explained solely by access to more training observations. We believe this is consistent with GGMP's design: repeated observations are compressed into local K-component mixtures, and once these are sufficiently well estimated, additional observations offer diminishing improvements. Our local-sample-size ablations in Section 4.4 support this interpretation. These results are now reported in Appendix D.5 of the revised manuscript.

Runtime on matched hardware: We thank the reviewer for highlighting this concern. All experiments were originally conducted on matched hardware, except for the additive-manufacturing MDN runs, which used a GPU-accelerated node rather than the standard CPU node. We included these results to demonstrate GGMP's computational efficiency even without GPU acceleration. Nevertheless, to ensure a direct and fair comparison, we have now rerun the MDNs on the same CPU hardware used for GGMP. For K = 1, 3, 5, 10, and 25, the MDN runtimes are 1802, 1947, 2221, 2493, and 2895 seconds, respectively, compared with GGMP runtimes of less than 5, 28, 47, 107, and 412 seconds. These results demonstrate that GGMP maintains a substantial wall-clock runtime advantage on the manufacturing benchmark even under matched hardware, and Appendix D.1 of the revised manuscript has been updated accordingly.

Manufacturing dataset: We deliberately selected this dataset to demonstrate GGMP on a real-world engineering application involving data collected and used by practitioners, rather than relying exclusively on synthetic or standard public benchmarks. Its 558,144 observations across 24 experimental conditions represent precisely the multivariate, low-N, high-T setting for which GGMP was designed. Although the number of distinct conditions is limited, we evaluate across three condition-level splits and three GGMP seeds, with additional paired comparisons and confidence intervals reported in Appendix D.4. The dataset will also be made publicly available alongside the final paper to enable independent reproduction of these results.

We thank the reviewer again for their constructive feedback. We hope the additional experimental evidence and clarifications address their concerns and would appreciate reconsideration of the rating.

Thank you for providing the additional experiment results, these solved my concern about the unequal comparison.

Thank you for taking the time to consider our additional experiments. We are glad that the results address your concerns. These results are now included in the revised manuscript.

We appreciate your feedback and hope the additional evidence will be reflected in your final assessment.

Reviewer_rcWqOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces the Generalized Gaussian Mixture Process (GGMP)​, a tractable GP method for repeated-measure conditional density estimation. It fits a local Gaussian mixture at each input, deterministically aligns components across inputs (sorting or MST–Hungarian matching), then trains one heteroscedastic GP per component with shared-weight optimization — avoiding the intractable K^N assignment structure while yielding a closed-form mixture predictive density.

Critical strengths

1/The paper's theoretical contribution is strong and self-consistent. It proposed the alignment-recovery guarantees (Theorem 3.3), the non-identifiability result at collisions (Proposition 3.4), and the consistency theorem (Theorem 3.5) together give a rigorous, stage-by-stage account of when the decoupled pipeline works and where it breaks. 2/Empirically, four controlled ablations are carefully designed and reported.

Critical weaknesses

1/The mathematic formulation is overly dense. 2/Sec. 4.3, The real-world additive-manufacturing dataset withheld until camera-ready, it cannot reproduce easily. As it is a general problem, why not just pick public dataset for verification. 3/ The contribution claims a "scalable GP solvers", however, only exact GPs are used, rather than any sparse GP. 4/Notation inconsistency: σ conflates variance and standard deviation; fitted quantities lack hats; within-component scale appears as both s and σ.

Questions for authors

See the above.

1 comment
AuthorsOpenReview ↗

Response to Reviewer rcWq

Update: The revised manuscript is now uploaded. We have expanded the intuitive explanations accompanying Assumption 3.2, Theorems 3.3 and 3.5, and the error decomposition in Section 3.4. We have also reformatted the assumptions and theorem statements to make their conditions and conclusions easier to follow. Section 3.3 now clarifies the scale notation and defines the alignment permutation pi_n, and Appendix C.1 distinguishes fitted covariance matrices. The abstract, contributions, and Sections 3.3.2 and 5 now clarify that we deliberately use exact GPs while retaining compatibility with sparse or inducing-point solvers. Section 4.3 and the reproducibility statement clarify the choice and planned release of the manufacturing dataset.

--

We sincerely thank the reviewer for the positive assessment of the paper. We respond to the queries and concerns below:

Mathematical presentation: We will revise the mathematical formulation to be more accessible, providing intuitive explanations alongside the formal statements. We will also clarify the interpretation of the assumptions while preserving the theoretical guarantees, which we appreciate the reviewer identifying as a key strength of the paper. We will also standardize the notation throughout to ensure that there’s a clear distinction between the variables everywhere in the paper, particularly with s and σ, and we will run another fine pass to check inconsistencies.

Choice and reproducibility of the additive manufacturing dataset: While we understand the concerns regarding the reproducibility, we would first like to emphasize that the anonymized dataset will be made publicly available alongside the final paper, enabling independent reproduction of the results. In addition, we deliberately selected this dataset over another public one to demonstrate the GGMP on a real world engineering problem involving data collected and used by practitioners. The dataset was relevant to us because it highlighted the intended setting, i.e. the low N, high T regime that the method is designed for. We believe this practical motivation is an important complement to our publicly reproducible synthetic and temperature experiments. The anonymized manufacturing dataset will also be made publicly available alongside the final paper, enabling independent reproduction of these results.

Compatibility with scalable GP solvers: We thank the reviewer for bringing this point to contention; we would like to clarify though, that our contribution is not a new sparse-GP method. Instead, an important feature of the GGMP is its modularity, where once the local mixture and components are aligned, the inference decomposes into independent standard GP regression problems. This permits the GP solver to be replaced with existing sparse or inducing point methods (or any other alternative), without changing the architecture. We deliberately chose exact GPs to isolate the contributions of local mixture fitting, component alignment, and variance modeling, without introducing additional approximation effects from the GP solver. We will make this distinction clearer in the manuscript.

We thank the reviewer again for their constructive feedback. We hope the clarifications and the revised manuscript address their concerns and would appreciate reconsideration of the rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.