- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper studies when robust fitting of a Gaussian process and Studentization can be combined without losing efficiency. Integrating out the shared variance scale, one gets a Student-t predictive instead of a Gaussian one, and this has a strictly positive gain in predictive quality that can be written in closed form. When the model is fitted with a density power divergence loss for robustness, the authors show that, for small robustness parameter, the gain shrinks linearly with this parameter, plus a small higher-order remainder. They also show that Studentizing any already fitted model can lower the predictive log likelihood only by a tiny amount. In experiments, the theory matches well for small robustness parameters, and Studentization applied to 3 existing robust GP methods generally improves the model.
Critical strengths
The paper is clearly written, carefully worded, well formated, and does not read like AI slop. All theoretical results seem correct and all empirical results seem believable. The claims are (mostly) demonstrated. Appendices are exhaustive and contain usefull information (though, to be honest, I did only read the proofs in the appendices).
Critical weaknesses
I have to bring the boring topic about noveltyapplicability. If this was an AISTATS, UAI or TMLR submission, I would clearly say "accept". My feeling is that the paper is at best borderline in novelty for ICLR. Hence, I vote weak reject. That being said, it would not be a mistake to accept the paper.
Points to highlight the novelty/applicability problem:
- The constants in the theorem are unevaluated and the interval is , which for n = 64 is about 0.004. The experiments mostly use value , outside it, so the theorem seems not easily applicable in practice.
- The setting is restrictive. It assumes fixed kernel hyperparameters, a shared variance scale, clean data etc.
- Effects are small, though consistent.
Minor points, which are easily fixable:
- The tables mit nats and 1e-3*nats as units. This confused me.
- Notation of K and is slightly inconsistent.
- The Boston housing dataset is discouraged from being used. Does it make sense to swap it?
Questions for authors
I have no questions.
If I should change my score, please address the novelty/applicability question.
We greatly appreciate your careful evaluation and constructive feedback. Our contributions are:
W1
The sufficient interval is narrow for our experimental settings because the proof uses conservative uniform bounds over training responses. Our result establishes that robust fitting and predictive efficiency can coexist, but does not provide a practical rule for selecting β (Section 6.1). The experiments (Figure 2 and Appendix F.2.3, Table 3) suggest that gain retention can hold over a wider range of β. Independent Monte Carlo estimates of gain retention exceeded 50% in all 446 conditions within the empirical cutoffs. Establishing less conservative guarantees is left for future work.
W2
fixed kernel hyperparameters
We acknowledge that our theoretical results, except for Proposition 1, assume fixed kernel hyperparameters. However, this is a common assumption in theoretical analyses of GP models (e.g., Bayesian optimization regret analysis).
shared variance scale
The shared variance scale preserves the conditional Gaussian model class of standard GP regression with i.i.d. Gaussian observation noise. The bijection (σf2,σϵ2)↦(τ,σ2)=(1/σf2,σϵ2/σf2) gives
=\tau^{-1}(\boldsymbol{K}+\sigma^2\boldsymbol{I}).$$ The kernel parameters and relative noise variance $\sigma^2$ can be estimated by empirical Bayes. Studentization accounts for scale uncertainty by integrating $\tau$ under the improper prior $\pi(\tau)\propto\tau^{-1}$ (Section 3.2). At $\beta=0$, under the clean model with the correct covariance shape held fixed, Studentization yields strictly lower predictive risk than the ordinary plug-in Gaussian predictor (exact gain $\Delta_{n,p}>0$; Lemma 1). ### __clean data__ This is a limitation; the clean-data analysis isolates the predictive cost of robust fitting. We complement this analysis with experiments using contaminated training data in the DPD setting (Section 6.3, Figure 3) and with existing robust GPs (Section 6.4, Figure 4 and Table 1). Motivated by these results, we are considering extending the theory to training observations drawn from a mixture $(1-\epsilon)P_{\mathrm{clean}}+\epsilon P_{\mathrm{contam}}$, with latent binary indicators assigning observations to the clean or contaminated component. ## W3 We agree that the observed PLL improvements are small. Studentization in GP prediction is a functional counterpart of the $t$-statistic: it accounts for uncertainty in the estimated scale while preserving the fitted mean. The correction requires no refitting, improves joint PLL in most tested conditions, and has a distribution-free bound on PLL decrease (Proposition 1). ## Minor - We standardized the table units to nats, removing the $10^{-3}$ rescaling (line 352). - We clarified the notation for $K$, $K_y=K+\sigma^2I$, and the predictive scale matrices (line 156). - We replaced Boston Housing with California Housing (line 454); the qualitative pattern of joint-PLL improvement is retained (Table 1).