acceptodds
Under review as a conference paper at ICLR 2027

Gaussian Process Latent Factor Regression for High-Dimensional Outputs

10 views

Abstract

In the sciences, regression tasks often require predicting high-dimensional outputs from few training examples. Multi-output Gaussian processes excel in low-data regimes but typically struggle with high-dimensional outputs. Compress-then-predict pipelines such as PCA-GP (principal component analysis plus Gaussian process regression) handle high dimensionality, but rely on bases optimized for reconstruction rather than prediction. To address this gap, we propose a method that represents each output as a linear-Gaussian decoding of a low-dimensional latent state drawn from a Gaussian process prior. By analytically marginalizing the decoder weights, we couple compression and prediction in a single objective that scales to high-dimensional outputs. We refer to this method as Gaussian process latent factor regression (GPLFR). We demonstrate GPLFR by building a spatially resolved emulator of global climate models for rocky exoplanets. Code: https://anonymous.4open.science/r/GPLFR-2C0B

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Emulation, Latent Variable Models, Multi-Output Regression, Surrogate Modelling, Gaussian Processes, Astrophysics, Climate Science

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_EgsMOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper tackles multi-output Gaussian processes in high-dimensional outputs. The idea presented in the paper hinges on representing each output as a linear-Gaussian decoding of a low-dimensional latent state drawn from a GP prior over the inputs. The authors demonstrate their method in toy examples as well as in biomedical optics and climate modelling for exoplanets.

Critical strengths

The most substantive strength of the paper is the rather simple idea combined with an interesting example study in exoplanet climate modelling. The supplement also contained further analysis and results about this, which I found interesting. To be frank, the paper might even be better received if it would be framed as an application paper in modelling the exoplanet data. However, then ICLR would probably not be the most impactful/sensible venue for such a study.

Critical weaknesses

The paper is nice overall, but considering the multitude of models already proposed in this space, I'm not fully convinced that the methodology alone in the paper makes it meet the bar for acceptance.

Questions for authors
  1. In addition to the simulation study and the demonstrations, did you also run this on low-dimensional example cases from other multi-output GP papers to gain some insights?

  2. The joint optimization problem is probably quite non-convex. In the experiments, did you have issues with the optimisation and hyperparameter sensitivity?

3 comments
AuthorsOpenReview ↗

Response to Reviewer EgsM

Thank you for the careful review and honesty on framing.

On novelty/framing.

It's true that this space holds many models; however, we do think our conception provides interesting points of novelty, although we agree it's less so in providing a new model class. In light of your comments (and another reviewer), we think that our framing didn't make our contributions clear enough and invited reading it as mainly a new model class.

We intend to reorder the introduction so that we first note how this model arises as a simplifying special case (albeit with modifications) of existing models and state our contributions more clearly (as below), before elaborating in Section 2.

  • we identify GPLFR as a modified special case of the supervised GPLVM lineage that is especially interesting for regression rather than the representation learning which the GPLVM literature tends to focus on
  • we give a practical method to fit this model that scales to high dimensions, and show it's a strong regression method and where it beats more familiar PCA-GP style methods, which are the standard default in the emulation problems we focus on (see e.g., Higdon+2008).
  • we show how this relates to the LMC, which is where most multi-output GP work has gone, through the alternative marginalization connection, which hasn't been noted as far as we know.

We kept the paper centred on the methodology because the same construction applies to other emulation problems (although yes a lot of the practical work was domain (exoclimate) oriented).

On questions.

  1. We didn't run on low-dimensional example cases simply because that is the regime where modelling capacity should go to output covariance, and LMC-like models are the natural choice. Although we agree it would be interesting to see where the balance of performance shifts between GPLFR-like and LMC-like models as dimensionality changes and will explore this.
  2. Yes we agree optimization is likely nonconvex. We do mention this in Section 5 and explore tempering sensitivity (which is the most important hyperparameter) in Appendix D.3.11. GPLFR without tempering still has the lowest RMSE on all eight variables of Table 1, so it is not overly sensitive there at least. In practice we didn't have major issues once the hyperparameters were chosen on small grids by cross-validation: fits are stable across training seeds (e.g., surface temperature RMSE 9.65 ± 0.10 K over five seeds) and cheap (under a minute on the exoclimate task). We will add a brief discussion of sensitivity to the other hyperparameters (learning rates, latent dimension) to Appendix B.4 (Posterior and MAP Estimation).

Thank you for your reply. Your plan for the intended changes to the manuscript appears like a good plan.

Thank you - we'd be be very happy if you think this might raise the papers informativeness score.

Reviewer_aka7OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes Gaussian Process Latent Factor Regression (GPLFR), an approach for predicting high-dimensional outputs by jointly learning a low-dimensional latent representation and GP-based regression through analytical marginalization of the decoder weights. The method is evaluated on synthetic and scientific tasks.

Critical strengths
  • The proposed formulation is relatively simple and mathematically tractable. In particular, the analytical marginalization of the decoder weights is interesting from a computational perspective, and the connection to LMC through alternative marginalizations of the same latent-factor model is clearly explained.
  • From the evaluation perspective, the propose method performs relatively good (I'm not an expert of climate models, so a defer the comment of the significance of these results to others)
Critical weaknesses

A few critical points, which basically boil down to novelty with respect to the literature (especially the one related to GPLVM).

  • The problem is that I struggle to identify any substantial novelty. You acknowledge that GPLFR reduces to a linear-decoder supervised GPLVM under certain assumptions. Marginalizing the weights of a linear-Gaussian decoder is also a standard construction in the GPLVM literature. I'd like a comment from the authors on this.

  • More generally, I think the paper misses a substantial part of the literature on GP latent variable models. In particular, I think you should consider Tran et al. (2023), Fully Bayesian Autoencoders with Latent Sparse Gaussian Processes (paper). This paper seems extremely closely related to your approach. They also use GP priors over latent representations indexed by observed inputs, together with a probabilistic decoder. If you check their generative model, it resembles yours. Not only that, their model additionally supports nonlinear decoders and sparse GP inference and a fully Bayesian treatment (which you discuss in Appendix B6). I think this deserves a more detailed comparison.

Questions for authors
  • I'm not sure I understand the justification for likelihood tempering. It is introduced to compensate for model misspecification (is it?), but is this just due to having a linear decoder or the structure of covariance? How much of the improvement comes from this regularization rather than the proposed modeling approach? Can you please comment on this?

  • The second question relates to the comparison to Tran et al. Can you explicitly show how the two probabilistic models relate, and whether GPLFR can be obtained as a special case of their framework by choosing a linear-Gaussian decoder? What changes in the inference procedure? I would also expect an experimental comparison with their approach, at least on a smaller dataset where both methods are computationally feasible, or a clear justification for why such a comparison is not appropriate.

4 comments
AuthorsOpenReview ↗

Response to Reviewer aka7 (1/2)

Thank you for the careful review and guidance.

On novelty/framing.

Yes it's true that marginalizing the weights is not a novel idea, and it's the main trick behind the dual PPCA construction of Lawrence+2005 that helps motivate GPLVMs (as a non-linearization of that dual PPCA). It is also true that under certain assumptions (an identity output covariance, a kernel shared across all latents, and no tempering), GPLFR can be seen as a linear-decoder supervised GPLVM.

But we do still think our conception is novel in that

  • we identify this special case as especially interesting for regression, which as far as we know hasn't been done. The GPLVM literature tends to focus on representation learning and nonlinear kernels (with the standard prior, a linear kernel just recovers PCA).
  • we give a practical method to fit this model that scales to high dimensions, and show it's a strong regression method and where it beats more familiar PCA-GP style methods, which are the standard default in the emulation problems we focus on (see e.g., Higdon+2008).
  • we show how this relates to the LMC, which is where most multi-output GP work has gone, through the alternative marginalization trick, which hasn't been noted as far as we know.

We understand that novelty is to some degree subjective, and we respect your opinion, but we would like to mention that other well-regarded methods in this area can also be seen as special cases of earlier ones. For instance: SLFM (Teh+2005) is a special case of the LMC in which each latent GP has its own kernel, introduced with an adapted but known sparse approximation that made fitting practical; the multi-task GP of Bonilla+2007 is the ICM with a practical fitting recipe and some experiments; and OILMM (Bruinsma+2020) is an LMC-type model restricted to an orthogonal mixing matrix (which makes exact inference much cheaper).

We do also think that in light of your comments we could have framed the paper better. It introduces GPLFR as a new method before saying where it comes from, which perhaps invites suspicion, and it leaves the GPLVM connection to the end of Section 2.2. We will reframe the introduction so that it starts from the supervised GPLVM and states our contributions more clearly (as stated above), before elaborating on that connection in Section 2.2.

Relation to Tran+2023.

We weren't aware of Tran+2023; thank you for pointing it out to us and we apologise for this oversight. We have read through it now and can agree it is related - it shares with GPLFR and supervised GPLVMs a GP prior on the latents over the inputs and a prior on the decoder. The difference is in the decoder and the inference schemes that follow primarily from that difference. Theirs is a neural network, which can't be integrated out, so they sample the decoder weights, latents and hyperparameters and train an encoder network alongside. Ours is linear, so we can integrate out the decoder analytically and there is no sampler or encoder. In the terms of our Section 2: the LMC integrates out the latents, GPLFR integrates out the decoder weights, and Tran+2023 integrate out neither.

Their work is also not as focused on very high-dimensional outputs, and there is reason to expect their approach to be harder to use there (with a fully connected decoder, as our non-image outputs require, the number of decoder weights grows with the output dimension, and they have to be sampled jointly with the latents).

We have now also run their method SGP-BAE on our exoclimate task, using the authors' code and a similar amount of tuning to our existing baselines around the authors' defaults (16 configurations by 3-fold cross-validation, five seeds). It performed poorly because we couldn't get the sampler to converge (e.g., surface temperature RMSE 17.3 K, against 9.7 K for GPLFR), even when fitting about 30x longer than GPLFR.

We think it's maybe not helpful to add this result to the paper or tune it more extensively as their method is more complex and part of the point of GPLFR is its simplicity, and it would be slightly unfair on their method without this more extensive adaptation, since their paper focused on lower dimensionalities (up to about dimensionality 1000, using a convolutional decoder). We will certainly add Tran+2023 and the associated GP-prior autoencoder literature (Casale+2018, Pearce 2020, etc.) to the related work.

AuthorsOpenReview ↗

Response to Reviewer aka7 (2/2)

Likelihood tempering.

The need for likelihood tempering comes more from the choice of covariance structure B\mathbf{B} than the fact of using a linear decoder itself. For our problems we set B=I\mathbf{B}=\mathbf{I}, so each output dimension is treated as conditionally independent (given latents Z\mathbf{Z}), but dimensions in outputs like fields are highly correlated so this overstates evidence about Z\mathbf{Z} in the likelihood and so can misbalance it relative to the prior. So yes, tempering is just a way to deal with this misbalancing (it can be motivated in terms of a generalized Bayesian update Bissiri+2016). We do look at how important tempering is in Appendix D.3.11 (Table D.10); setting β=1\beta=1 changes the exoclimate RMSE by about 2% at most on any variable. GPLFR without tempering still has the lowest RMSE on all eight variables of Table 1. Tempering is more important for calibration (energy scores are notably better on all variables and SSR is better on average). We will also add this ablation for the synthetic study.

Sorry for the last-minute response. Thanks for the reply to my questions, I have a clear picture. To me the paper still remains borderline but I will discuss it with the other reviewers.

Fair enough. We hope that at least with these changes the paper is more informative.

Reviewer_SsuyOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Gaussian Process Latent Factor Regression (GPLFR), a probabilistic framework for high-dimensional multi-output regression in low-data regimes. Addressable to the mismatch in traditional "compress-then-predict" pipelines (e.g., PCA-GP, where compression is purely optimized for reconstruction rather than input-dependent prediction), GPLFR aims to learn latent representations and linear-Gaussian decoding jointly. Technically, the approach marginalizes the linear decoder weights analytically to form a collapsed likelihood, making the model mathematically equivalent to a dual form of the Linear Model of Coregionalization (LMC) or a linear-decoder variant of Supervised GPLVM. The authors validate GPLFR on synthetic data, a 500-dimensional biomedical optics task, and an exoplanet climate emulation benchmark.

Critical strengths

Principled Probabilistic Formulation: The mathematical connection between GPLFR and LMC (as two distinct marginalizations of the same joint linear-Gaussian process) is clean, elegant, and well-formalized.

Real-world Application Domain: The exoplanet climate emulation task addresses a meaningful scientific domain with structured missing data and multi-simulator integration challenges.

Critical weaknesses

Limited Novelty & Dated Paradigm: Marginal technical advance over classic GP literature (e.g., dual LMC, linear-supervised GPLVM). Relying on linear decoders for high-dimensional fields lacks algorithmic breakthroughs for ICLR [cite: 18, 25].

Optimization Brittleness: Joint optimization is non-convex and heavily reliant on ad-hoc heuristics (likelihood tempering β\beta, latent noise λ\lambda) to prevent mode collapse.

Weak Baselines & Marginal Empirical Gains: Lacks comparison with modern neural surrogates (e.g., Fourier Neural Operators) for high-dimensional field emulation [cite: 25]. Performance gains over simple baselines (e.g., PPCA-ICM) are marginal.

GPLFR presents a mathematically coherent unification of latent GP regression and linear decoding. However, its core methodological novelty is incremental relative to classic GP literature, its optimization relies on heuristic tempering, and the empirical evaluation fails to compare against contemporary deep learning surrogates for scientific field emulation.

Questions for authors

How does GPLFR perform against parameter-efficient or small-data fine-tuned Neural Operators (e.g., Fourier Neural Operators / Spherical Neural Operators) on the exoplanet climate dataset?

How sensitive are the predictive variance and calibration metrics (Energy Score/SSR) to the choice of the inverse temperature parameter β\beta? Is there an automated way (e.g., cross-validation or marginal likelihood) to select β\beta without manually searching?

What is the computational scaling limit of the collapsed likelihood implementation when both NN (sample size) and DyD_y (output dimensions) scale simultaneously?

1 comment
AuthorsOpenReview ↗

Response to Reviewer Ssuy

  • Novelty and 'dated paradigm'. We can't really address your novelty or 'dated paradigm' qualms since you only cite the papers with numbers that don't link to anything. We don't think it's fair to label Gaussian processes or methods with linear decoders (we're not sure which you mean) a 'dated paradigm' since they are still the strongest methods on many low-data problems, some of which we show here.
  • Neural operators. We don't think including Fourier neural operators would improve this paper. Neural operators learn maps between fields, whereas our inputs are collections of scalars, so they would require adaptation. The two neural baselines we do include have much higher RMSE than GPLFR (see Table 1), and we would expect an adapted neural operator to not perform that much better. Also this paper is intended as a focused study on jointly learning compression and regression in low-data, high-dimensional-output settings like scientific emulation, not as an extensive benchmark.
  • 'Marginal' gains over a 'simple' baseline. We don't think it's fair to say the gains over PPCA-ICM are 'marginal' or that PPCA-ICM is 'simple' – it's the natural compress-then-predict comparator for GPLFR and GPLFR isn't clearly more complicated than PPCA-ICM. You may be confusing simpler with more familiar.
  • Computational scaling. This is noted in Appendix B.3.
  • Tempering as an 'ad-hoc heuristic' that GPLFR is 'heavily reliant' on, and energy score/SSR sensitivity to tempering parameter β\beta. Likelihood tempering has a decision-theoretic justification as a generalized Bayesian update. β\beta does matter but GPLFR isn't completely reliant on it: without tempering (β=1\beta=1) the RMSE changes by about 2% at most on any variable. We show the β\beta sensitivity for energy score and SSR (alongside RMSE) for the exoclimate task in Table D.10.
  • Choosing β\beta automatically. You ask whether β\beta could be selected by cross-validation. It already is: we choose it by 3-fold cross-validation over the grid in Table D.6. Interpreting this charitably, yes β\beta introduces an additional tuning burden and better heuristics for choosing it would reduce this cost, which we acknowledge in the paper.
  • Presentation and soundness scores. We don't really see why this is getting a 2 in presentation and soundness, and you haven't named a passage that was hard to follow or any errors.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.