acceptodds
Under review as a conference paper at ICLR 2027

Reconciling robustness and finite-sample efficiency in Gaussian process regression

Abstract

Robust fitting reduces the influence of outliers but can sacrifice finite-sample predictive efficiency when the data follow the assumed model. In this work, we quantify how robust fitting changes the finite-sample predictive gain from Studentization, which accounts for uncertainty in an estimated variance scale rather than treating its estimate as known. Under the assumed Gaussian process (GP) model with a shared variance scale, fixed kernel parameters, and more than two training observations, Studentization yields a strictly positive finite-sample gain in expected log predictive density over plug-in Gaussian prediction, and this gain is available in closed form. We combine Studentization with density power divergence (DPD) fitting and quantify the predictive cost of robustification. Our analysis relates this cost to model complexity and the influence of the training responses on test predictions. It gives a sufficient condition for retaining a prescribed fraction of the exact Studentization gain over ordinary Gaussian prediction. Controlled synthetic experiments support the analysis and show predominantly positive gains from Studentization in prediction. Beyond the theoretical assumptions, applying Studentization to existing robust GP methods mostly improves joint predictive log likelihood, with only small decreases otherwise. Finally, we derive a distribution-free bound on the possible decrease in log predictive density when Studentization is applied to an already fitted model. The code is available at https://anonym ous.4open.science/r/rsgpr-0E17/.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Gaussian process, Student-t distribution, robust inference, variational inference

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_oW7JOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies when robust fitting of a Gaussian process and Studentization can be combined without losing efficiency. Integrating out the shared variance scale, one gets a Student-t predictive instead of a Gaussian one, and this has a strictly positive gain in predictive quality that can be written in closed form. When the model is fitted with a density power divergence loss for robustness, the authors show that, for small robustness parameter, the gain shrinks linearly with this parameter, plus a small higher-order remainder. They also show that Studentizing any already fitted model can lower the predictive log likelihood only by a tiny amount. In experiments, the theory matches well for small robustness parameters, and Studentization applied to 3 existing robust GP methods generally improves the model.

Critical strengths

The paper is clearly written, carefully worded, well formated, and does not read like AI slop. All theoretical results seem correct and all empirical results seem believable. The claims are (mostly) demonstrated. Appendices are exhaustive and contain usefull information (though, to be honest, I did only read the proofs in the appendices).

Critical weaknesses

I have to bring the boring topic about noveltyapplicability. If this was an AISTATS, UAI or TMLR submission, I would clearly say "accept". My feeling is that the paper is at best borderline in novelty for ICLR. Hence, I vote weak reject. That being said, it would not be a mistake to accept the paper.

Points to highlight the novelty/applicability problem:

  • The constants in the theorem are unevaluated and the interval is ≤1/(4∗(n+1))\le 1/(4*(n+1)), which for n = 64 is about 0.004. The experiments mostly use value ≥0.01\ge 0.01, outside it, so the theorem seems not easily applicable in practice.
  • The setting is restrictive. It assumes fixed kernel hyperparameters, a shared variance scale, clean data etc.
  • Effects are small, though consistent.

Minor points, which are easily fixable:

  • The tables mit nats and 1e-3*nats as units. This confused me.
  • Notation of K and KfK_f is slightly inconsistent.
  • The Boston housing dataset is discouraged from being used. Does it make sense to swap it?
Questions for authors

I have no questions.

If I should change my score, please address the novelty/applicability question.

1 comment

We greatly appreciate your careful evaluation and constructive feedback. Our contributions are:

  1. A finite-sample analysis quantifying the predictive cost of robust fitting relative to the exact Studentization gain and establishing a sufficient condition for retaining that gain.
  2. A distribution-free bound on the PLL decrease from Studentization, applicable also to robust GPs with learned kernels.

W1

The sufficient interval is narrow for our experimental settings because the proof uses conservative uniform bounds over training responses. Our result establishes that robust fitting and predictive efficiency can coexist, but does not provide a practical rule for selecting β\beta (Section 6.1). The experiments (Figure 2 and Appendix F.2.3, Table 3) suggest that gain retention can hold over a wider range of β\beta. Independent Monte Carlo estimates of gain retention exceeded 50% in all 446 conditions within the empirical cutoffs. Establishing less conservative guarantees is left for future work.

W2

fixed kernel hyperparameters

We acknowledge that our theoretical results, except for Proposition 1, assume fixed kernel hyperparameters. However, this is a common assumption in theoretical analyses of GP models (e.g., Bayesian optimization regret analysis).

shared variance scale

The shared variance scale preserves the conditional Gaussian model class of standard GP regression with i.i.d. Gaussian observation noise. The bijection (σf2,σϵ2)↦(τ,σ2)=(1/σf2,σϵ2/σf2)(\sigma_f^2,\sigma_\epsilon^2)\mapsto(\tau,\sigma^2)=(1/\sigma_f^2,\sigma_\epsilon^2/\sigma_f^2) gives

=\tau^{-1}(\boldsymbol{K}+\sigma^2\boldsymbol{I}).$$ The kernel parameters and relative noise variance $\sigma^2$ can be estimated by empirical Bayes. Studentization accounts for scale uncertainty by integrating $\tau$ under the improper prior $\pi(\tau)\propto\tau^{-1}$ (Section 3.2). At $\beta=0$, under the clean model with the correct covariance shape held fixed, Studentization yields strictly lower predictive risk than the ordinary plug-in Gaussian predictor (exact gain $\Delta_{n,p}>0$; Lemma 1). ### __clean data__ This is a limitation; the clean-data analysis isolates the predictive cost of robust fitting. We complement this analysis with experiments using contaminated training data in the DPD setting (Section 6.3, Figure 3) and with existing robust GPs (Section 6.4, Figure 4 and Table 1). Motivated by these results, we are considering extending the theory to training observations drawn from a mixture $(1-\epsilon)P_{\mathrm{clean}}+\epsilon P_{\mathrm{contam}}$, with latent binary indicators assigning observations to the clean or contaminated component. ## W3 We agree that the observed PLL improvements are small. Studentization in GP prediction is a functional counterpart of the $t$-statistic: it accounts for uncertainty in the estimated scale while preserving the fitted mean. The correction requires no refitting, improves joint PLL in most tested conditions, and has a distribution-free bound on PLL decrease (Proposition 1). ## Minor - We standardized the table units to nats, removing the $10^{-3}$ rescaling (line 352). - We clarified the notation for $K$, $K_y=K+\sigma^2I$, and the predictive scale matrices (line 156). - We replaced Boston Housing with California Housing (line 454); the qualitative pattern of joint-PLL improvement is retained (Table 1).
Reviewer_5CtVOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper observes a simple but interesting operation: replacing the Gaussian distribution used in GP prediction with a Student-t distribution may improve predictive probabilistic quality. Building on this observation, the authors theoretically derive the exact gain of Studentization over Gaussian prediction and its joint dependence decomposition, then further derive the first-order effect of DPD robustification on the predictive gain, and finally prove a uniform bound on the worst-case decrease in predictive log likelihood when replacing Gaussian prediction with Student-t prediction. The experimental results support these conclusions.

Critical strengths
  1. Clear and meaningful research problem

This paper focuses on the trade-off between robust training and finite-sample predictive efficiency, presenting a clear research problem. Studentization can be directly applied to existing Robust GP models without retraining, and it improves joint predictive log likelihood under most experimental conditions, making it potentially valuable in practical applications.

  1. Solid theoretical analysis

This paper contains extensive theoretical analysis and proofs, which are generally rigorous. It not only quantifies the predictive cost of robustification but also relates it to model complexity and the influence of training data on test predictions. The theoretical analysis and experiments also follow a coherent logic: the authors first validate the risk expansion under controlled conditions, then introduce outliers to distinguish the contributions of robust training and Studentization.

Critical weaknesses
  1. Unclear parameter selection

The theoretical proofs in the paper appear to impose restrictions on the parameters, but the current theoretical guarantees are also difficult to use to guide practical parameter selection. In the experiments, the authors introduce β values (β∈{0.01,0.05,0.1}\beta \in \{0.01, 0.05, 0.1\}) without explaining why these values were chosen or demonstrating that they satisfy the parameter restrictions required by the theoretical proofs.

  1. Incomplete experimental evaluation

When the same scale matrix is maintained, the Student-t distribution not only changes the tail shape and joint dependence structure of the distribution but also increases the actual predictive covariance. The paper does not compare its approach with simple Gaussian variance scaling (e.g., a Gaussian distribution with the same covariance). Such a comparison would help provide stronger evidence for the authors' claim that replacing the Gaussian predictions of existing Robust GP models with Student-t predictions can improve joint PLL.

Questions for authors
  1. Please provide further explanations for the parameter choices in the experiments, particularly the rationale for selecting β∈{0.01,0.05,0.1}\beta \in \{0.01, 0.05, 0.1\}.

  2. Please include additional experiments comparing their method with simple Gaussian variance scaling.

  3. Please carefully check their submission again, as the paper currently contains some minor errors. For example, the code link in the abstract section of the submission page contains an obvious error, with an extra space in the URL.

3 comments

We sincerely appreciate your careful review and constructive suggestions.

W1, Q1

We explored β\beta on a fixed grid to assess both the local expansion and behavior at larger values (Section 6.1). The theorem uses conservative uniform bounds to establish that robust fitting and predictive efficiency can coexist, but does not certify all tested β\beta values or provide a practical selection rule. Our experiments suggest gain retention over a wider range (Figure 2 and Appendix F.2.3, Table 3). Establishing less conservative guarantees is left for future work.

W2, Q2

We added a comparison with a covariance-matched Gaussian (Section 6.4; Appendix G.4, Table 13). Studentized predictions still improved joint PLL in most conditions for all three methods, suggesting that tail shape also matters for joint prediction.

ModelPositive conditionsMedian ΔPLL/p\Delta\mathrm{PLL}/p (nats)Minimum ΔPLL/p\Delta\mathrm{PLL}/p (nats)
RCGP15/160.021568-0.000927
RP14/160.001480-0.000633
MGP13/160.016987-0.000777

Q3

Thank you for your careful reading. We corrected the extra space in the code URL on the submission page and addressed the presentation issues, including those pointed out by Reviewers 16DP and oW7J. We will check the submission again before the camera-ready version.

Thank you for your response and the additional experiments. The new comparison with covariance-matched Gaussian predictions is helpful, and I appreciate the corrections to the minor errors in the submission. I have decided to maintain my original rating of 3.

Thank you very much! We believe that the additional experiments suggested in your review clearly demonstrate the effectiveness of our work.

Reviewer_16DPOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates combined use of robust fitting (with density power divergence) and 'Studentisation' in Gaussian process regression. The former is aimed at reducing the impact of outliers on model fitting and the latter at improving the predictive density. Whilst robust fitting can reduce the adverse influence of outliers it comes at the expense of reduced predictive performance when the data follow the assumed model. The paper shows that if the GP model incorporates a shared variance scale and if the data is generated according to this GP model, then Studentization provides a strictly positive finite-sample gain in expected log predictive density over a plug-in Gaussian predictor; this gain is derived in closed form. The closed form reveals how the gain depends on pp , the number of joint predictions made. It is the joint-prediction setting in which several results in the paper become particularly relevant. Given various assumptions, Theorem 1 of the paper shows how the gain in expected log predictive density degrades if robust fitting is applied when the data adheres to the outlier-free GP model assumptions (i.e. when outlier protection is not needed). A distribution-free bound is also derived for the predictive log likelihood decrease that can arise when Studentization is applied to an already fitted model. A range of experiments are carried out to support the analysis as well as to assess the potential benefit of applying Studentization after fitting.

Critical strengths

S1. The paper presents new theory with explicit formulas to provide useful insights into the tradeoff between robust fitting and predictive performance (the latter in the well specified case).

S2. The experiments on synthetic data are well chosen and support the theory. The experiments on real data provide evidence that post-fitting Studentisation can often be beneficial in practice.

Critical weaknesses

W1. The main impact of the paper's results appears to centre around small datasets with relatively large number pp of joint predictions. The main paper results also apply specifically to a GP model with a shared variance factor plus associated choice of prior. To avoid criticism that this is a niche area the paper should identify important applications where this combination of circumstances and model choice applies.

W2. Some areas of the paper would benefit from expanded justification - see my questions below

Questions for authors

(1) Can you identify application areas where the central results of the paper are particularly relevant (see W1 above).

(2) Can you explain in simple terms what assumptions the solution path constructed in Appendix E.2 rests upon and justify why it is reasonable to assume that these capture the behaviour of unconstrained DPD fitting (it would also be helpful if such an explanation appeared in the main text). Can you also provide some additional justification for the particular choice you have adopted for the predictor in 4.2, including the choice to retain the Gamma shape through the KL projection - were other options considered but ruled out?

(3) Apart from the squared exponential (RBF) kernel, did you consider looking at other choices of kernel for the synthetic experiments. The coefficients in the expansion of equation 10, Theorem 1 are kernel dependent. The RBF kernel models very smooth processes whereas, for example, the Matern ν=\nu = 1/2 kernel models rough process (and ν=3/2,5/2,…\nu = 3/2, 5/2,\dots between these). Might the modeling of both rough and smooth processes shed some light on how process-smoothness/kernel-choice impacts the predictive performance degradation arising from robust fitting?

Some presentational issues:

  • It would be helpful if the term $ I_{n,p}$ were defined in lemma 1 (the proof in appendix C.2 expresses it as a KL divergence and this KL expression for In,pI_{n,p} could be given explicitly in lemma 1).
  • It would help the reader's interpretation of figure 4 if the caption were to explain that vertical jitter in the 9 plotted rows is (I believe?) purely to provide visual separation, i.e. vertical position within each row has no meaning.
4 comments

We deeply appreciate your careful reading and insightful comments.

W1, Q1

The shared-scale formulation preserves the conditional Gaussian model class of standard GP regression with i.i.d. Gaussian observation noise. Specifically, for any positive signal and noise variances (σf2,σϵ2)(\sigma_f^2,\sigma_\epsilon^2), the one-to-one change of variables (σf2,σϵ2)↦(τ,σ2)=(1/σf2,σϵ2/σf2)(\sigma_f^2,\sigma_\epsilon^2)\mapsto(\tau,\sigma^2)=(1/\sigma_f^2,\sigma_\epsilon^2/\sigma_f^2) gives

=\tau^{-1}(\boldsymbol{K}+\sigma^2\boldsymbol{I}).$$ The kernel parameters and relative noise variance $\sigma^2$ can be estimated by empirical Bayes. Studentization accounts for uncertainty in the overall scale by integrating $\tau$ under the improper prior $\pi(\tau)\propto\tau^{-1}$ (Section 3.2). At $\beta=0$, under the clean model with the correct covariance shape held fixed, this Studentized predictor achieves strictly lower predictive risk than the ordinary plug-in Gaussian predictor (exact gain $\Delta_{n,p}>0$; Lemma 1). Relevant application areas include materials science and biomedical engineering, where limited physical measurements often need to support predictions at many unobserved conditions or locations. Specific examples include: - Material assessment: Geyer et al. (2021) predict a soil-resistance profile over approximately 6.4 m from $n=13$ measurements, and a concrete-strength field over a 105 m × 8 m wall from $n=24$ measurements. Their predictions account for scale uncertainty. - Retinal-implant calibration: Sadeghi and Beyeler (2025) predict unmeasured electrode thresholds for implant calibration. For example, $n=20$ measured thresholds support predictions at $p\le40$ remaining locations in a 60-electrode array. Perceptual reports are affected by fatigue and attention. - Functional electrical stimulation calibration: Onoue et al. (2026) use around $n=50$ observations to predict motor-response distributions across $p=496$ electrode pairs and model subject-specific effective stimulation sites. They used studen-t likelihood to address physiological variability. These examples motivate combining finite-sample uncertainty quantification with robust fitting when measurements are costly and potentially unreliable. We added these motivation to the Introduction. References: - Geyer, S., Papaioannou, I., and Straub, D. (2021). "Bayesian analysis of hierarchical random fields for material modeling." *Probabilistic Engineering Mechanics*. - Sadeghi, R., and Beyeler, M. (2025). "Efficient Spatial Estimation of Perceptual Thresholds for Retinal Implants via Gaussian Process Regression." *IEEE EMBC 2025*. - Onoue, K., Endo, S., Ikeda, K., and Hirche, S. (2026). "Robust Bayesian Optimization for Reliable Functional Electrical Stimulation Calibration." *IEEE EMBC 2026*. ## W2, Q2 __solution path__ The solution-path construction uses the following conditions: - Fixed training inputs and a fixed positive definite kernel matrix. - A fixed positive relative noise variance. - A nonzero training response vector. - A sufficiently small robustness parameter, $0\le\beta\le b_0$, as specified in Theorem 1. Under these conditions, Appendix E.2 proves the existence of a smooth path of strict local minima of the original unconstrained DPD objective starting from the Gaussian fit at $\beta=0$. This justifies local continuation of unconstrained DPD fitting from the Gaussian solution, consistent with our clean-data optimization protocol (Appendix F.1). We added this explanation to Section 5. As discussed in our responses to Reviewer 5CtV (W1/Q1) and Reviewer oW7J (W1–W2), the theory establishes the existence of solutions retaining predictive gains under an idealized, uncontaminated GP model. We complement this analysis with experiments under contamination, the practical motivation for robust fitting (Sections 6.3–6.4, Figures 3–4 and Table 1). __why DPD predictor__ We chose DPD as a classic robust divergence that keeps robust variational inference simple and analytically tractable (Section 4.1). - It is a one-parameter member of the broad $\alpha$–$\beta$ divergence family. - It controls robustness through density exponentiation, with ordinary GP fitting recovered in the KL limit. - In our setting, both the expected loss and the stationarity conditions are available analytically. __constrained KL projection__ We prioritized consistency with standard Studentization based on the number of training observations (Section 4.2). - The GP still uses all $n$ training points. We update the mean precision to a robust estimate while retaining the Gamma shape $n/2$ and relative variance, $\operatorname{Var}(\tau)/\{\mathbb E[\tau]\}^{2}=2/n$. - Given the reference Gamma posterior and the robust mean-precision constraint, this shape follows from the KL projection (Appendix B). - Adjusting the shape using effective degrees of freedom is a possible alternative, which we leave for future work.

W2, Q3

We added comparisons of RBF with Matérn kernels of smoothness 1/21/2, 3/23/2, and 5/25/2 (Section 6.2; Appendix F.2.4, Figure 6 and Table 4).

Gain retention at small β\beta also holds empirically for these Matérn kernels. At n=64,p=16,β=0.01n=64,p=16,\beta=0.01, retention ranges from 93.3% to 98.3% across the four kernels. At β=0.05\beta=0.05, the differences widen: retention is 81.2% for RBF and 38.3% for Matérn 1/21/2.

The coefficient of the first-order cost, c1,pc_{1,p}, depends on the kernel and input design. We also give a theoretical counterexample showing that it need not be monotone in Matérn smoothness (Appendix F.2.4).

All results below use p=16p=16. Coefficients are in nats; retained gains are 100Gp(β)/Δn,p100G_p(\beta)/\Delta_{n,p} at n=64n=64, with one Monte Carlo standard error in parentheses.

Kernelc1,pc_{1,p}, n=8n=8c1,pc_{1,p}, n=64n=64Retained gain (%), β=0.01\beta=0.01Retained gain (%), β=0.05\beta=0.05Retained gain (%), β=0.1\beta=0.1
Matérn 1/20.3333600.10611793.3 (0.2)38.3 (1.0)-97.2 (3.0)
Matérn 3/20.3801200.04874296.9 (0.1)70.2 (1.0)2.3 (4.5)
Matérn 5/20.3882280.03660097.5 (0.1)74.7 (1.0)16.5 (3.8)
RBF0.3553810.02352998.3 (0.1)81.2 (0.8)37.3 (2.9)

Minor

  • We added the explicit KL divergence expression for the dependence term In,p\mathcal I_{n,p} to Lemma 1, as suggested (line 284).
  • We replaced the jittered display in Figure 4 with box plots to make the distributions across splits easier to interpret (line 447).

Thank you for the detailed responses to my questions and corresponding pdf updates. On the basis of your answers and clarifications I am happy to maintain my rating of 3 . If this paper fails to be accepted for ICLR then I nevertheless feel it has much to offer and here I share reviewer oW7J's view that it may have a stronger case for acceptance at UISTATS, UAI or TMLR where the novelty and wider impact angle may not be such a strong determining factor. I wish you all the best with publication of your paper.

Thank you so much for your kind words and insightful comments! They surely helped us to think deeper about our own work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.