acceptodds
Under review as a conference paper at ICLR 2027

CA-MFPD: Covariance-Aware Multi-Fidelity Posterior Distillation under Scarce High-Fidelity Data

Abstract

Multi-fidelity regression uses cheap, biased low-fidelity data to reduce the need for expensive high-fidelity evaluations. Gaussian process (GP) models provide joint predictive distributions, but repeated GP inference can be costly. We propose covariance-aware multi-fidelity posterior distillation (\method), which trains a neural surrogate model to reproduce the teacher's predictive distribution. We use a residual multi-fidelity GP to predict high-fidelity outputs at sampled inputs, then adjust predictive uncertainty using leave-one-out (LOO) errors on high-fidelity data. During training, a joint Gaussian KL loss brings the student's predictive mean and covariance closer to the teacher's. The student represents its covariance as a low-rank term plus a diagonal term. The off-diagonal entries represent correlations between inputs. We also compute the student's mean error at observed high-fidelity points and include it in the training objective. Our analysis shows that joint KL compares correlations between inputs, which pointwise comparison omits. We bound batch KL error in terms of mean and covariance approximation errors, and show that a lower expected batch KL gives a tighter bound on average pairwise covariance error. Numerical results show that CA-MFPD has lower overall prediction and coverage errors and faster repeated inference than standard co-kriging, with competitive results against nonlinear GP and neural multi-fidelity baselines.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Multi-fidelity regression, Covariance-aware posterior distillation, Data augmentation, Gaussian processes, Neural surrogate models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ZxYMOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to predict high-fidelity (HF) outputs when many low-fidelity (LF) observations and only a few HF observations are available. The proposed model CA-MFPD first fits a Gaussian process (GP) to the LF data. It then fits a second GP to the difference between the HF observations and the adjusted LF predictions. Together, these models form a teacher that predicts HF outputs and their uncertainty. Next, they use leave-one-out errors on the HF data to adjust that uncertainty. They then train a neural student to learn the teacher's predictive distribution. Real HF observations also guide the student's predicted values. An optional HF-only GP can help at prediction time. The paper analyzes this approach and evaluates it on ten benchmark problems, comparing prediction accuracy, uncertainty and computational cost with other methods.

Critical strengths
  1. The CA-MFPD takes joint predictions from a multi-fidelity GP, calibrates their uncertainty using scarce HF data and trains a student from them.

  2. The empirical study is substantial and relatively transparent about its tradeoffs: ten benchmarks, 20 seeds, seven nested HF budgets, paired comparisons, uncertainty metrics, ablations and inference timings.

Critical weaknesses

The paper's distinctive empirical claim is covariance-aware distillation, but the bencemark evaluation reports only the point accuracy and marginal uncertainty. Figure 4 illustrates the covariance issue with a two-point toy example, but the paper does not directly report how closely the student matches the teacher’s cross-input covariance on held-out benchmark inputs. That makes it harder to judge whether the extra joint training helps for its stated purpose.

Questions for authors

Please give the precise weighted-ridge objective and weights for fitting (b,ρ)(b,\rho), the ridge value and the GP hyperparameter fitting procedure. These choices affect both the teacher covariance and its LOO calibration.

4 comments
AuthorsOpenReview ↗

Response to Reviewer ZxYM

Dear Reviewer,

Thank you for your careful review and for recognizing the method's use of joint multi-fidelity GP predictions, HF-based calibration, and the scope of the empirical study. We address the main weakness and the methodological question below.

W1. Direct evaluation of cross-input covariance. We agree that the two-point example in Fig. 4 illustrates why marginal matching cannot identify cross-input covariance, but does not by itself show how closely the trained student matches the teacher on benchmark inputs. We added a matched comparison of joint Gaussian KL and marginal Gaussian KL across six benchmarks, three nested HF budgets per benchmark, and five paired seeds. The variants share the teacher, data splits, calibration, student architecture and rank, initialization, Sobol augmentation points, loss weights, optimizer, training budget, and HF anchors. Marginal Gaussian KL matches the teacher and student means and marginal variances at each input but omits cross-input covariance.

MetricTypeJoint KL lower (interval width: narrower)
NRMSEPointwise prediction9/18
NLLPointwise prediction12/18
CRPSPointwise prediction9/18
90% coverage errorPointwise prediction7/18
90% interval width (narrower)Pointwise prediction13/18
Relative covariance Frobenius errorCross-input dependence15/18
Off-diagonal correlation RMSECross-input dependence18/18
Batch-mean variance errorCross-input dependence15/18

The batch-mean metric evaluates predictive variance for an equal-weight average over a held-out input batch. These results separate pointwise predictive performance from direct cross-input dependence measures.

Q1. Weighted-ridge fit for (b,ρ)(b,\rho) and GP hyperparameter fitting. At HF inputs, let Ai=(1,μL(xiH))A_i=(1,\mu_L(x_i^H)), let AA be the matrix with rows AiA_i, let wi=(vL(xiH)+10−10)−1w_i=(v_L(x_i^H)+10^{-10})^{-1}, and set W=diag(wi)W=\mathrm{diag}(w_i). We estimate (b,ρ)(b,\rho) by minimizing ∑iwi(yiH−b−ρμL(xiH))2+λρρ2\sum_i w_i(y_i^H-b-\rho\mu_L(x_i^H))^2+\lambda_\rho\rho^2, where λρ=10−6max⁡{12tr(A⊤WA),10−12}\lambda_\rho=10^{-6}\max\{\tfrac12\mathrm{tr}(A^\top W A),10^{-12}\}. The intercept is unpenalized, and the fitted slope is clipped to [−5,5][-5,5] before fitting the discrepancy GP.

The exact GPs use a constant-scaled ARD Matérn-5/25/2 kernel, normalized targets, and a fixed diagonal nugget of 10−610^{-6}. The signal-variance initialization is 1 with bounds [10−6,106][10^{-6},10^6]; each length scale is initialized at 0.2 with bounds [0.02,2][0.02,2]. We maximize the exact log marginal likelihood using L-BFGS-B, with eight restarts for the main fits and one restart for each HF-LOO fit. For each HF observation, the LF GP is held fixed, and the affine and discrepancy components are refit after removing that observation; the resulting LOO residuals determine the covariance scale. These implementation details are now specified in the reproducibility appendix.

Thank you for pointing out the need for a direct benchmark-level covariance audit and more precise fitting details. These additions make the empirical test of the central claim and the model-fitting procedure clearer.

Dear Reviewer,

Thank you for your comments on the contribution and experimental evaluation of our paper. We have added an explanation of how multi-fidelity posterior transfer can be used in a cross-input engineering task.

We evaluate near-hole stress screening on SimuStruct perforated plates. For each geometry, we fit the models using a coarse-mesh low-fidelity stress field and six high-fidelity stress measurements. We then draw 4,000 samples from the student's full joint predictive distribution over near-hole locations to estimate the probability that the maximum near-hole von Mises stress exceeds 585.31 MPa.

We compute the class-balanced Brier score by first calculating the mean Brier score separately for exceedance and non-exceedance cases, and then averaging the two class-specific scores with equal weight. For each plate, we first average the predicted probabilities across five seeds. The results are as follows:

Number of plate geometriesExceedance casesJoint KLMarginal KL
102/100.04720.0941
95/90.15400.1699
10028/1000.23000.2655

Among the 100 additional geometries, joint KL detects 16 of the 28 exceedance cases, compared with 10 for marginal KL.

This experiment illustrates one use of multi-fidelity posterior transfer in spatial screening: using a small number of high-fidelity measurements and a low-fidelity stress field to predict the joint stress distribution across multiple near-hole locations, and then estimate the probability of a spatial maximum exceeding the threshold.

Thank you again for your valuable comments.

Thank you for providing the new ablation results and these addresses my concern about direct covariance evaluation. I'm willing to raise the score.

Thank you for your thoughtful follow-up and for reconsidering your score. We are glad that the additional ablation results and direct covariance evaluation addressed your concern. We appreciate your careful reading and constructive feedback, which helped us strengthen the paper.

Reviewer_DUzAOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CA-MFPD, which distills a multi-fidelity Gaussian process into a neural surrogate model for regression under scarce high-fidelity data. This approach aims to transfer both predictions and cross-input covariance from the GP teacher, while avoiding the high cost of repeated GP inference. The experimental results show that CA-MFPD achieves lower prediction and coverage errors than co-kriging with faster repeated inference.

Critical strengths

The paper is well-presented and easy to follow.

The paper tackles covariance-aware GP distillation with MLP surrogate for the multi-fidelity setup, improving sampling efficiency.

The proposed method grants more calibrated uncertainty than co-kriging.

Critical weaknesses

The novelty is very limited. Although the core idea is multi-fidelity posterior distillation, the multi-fidelity teacher model is the standard co-kriging model, and the neural posterior student closely follows [1] and [2], which are already well-explored literatures.

The efficiency improvement also requires more support. The training of CA-MFPD takes ~65s while co-kriging only takes ~1.8s. Although the authors show that CA-MFPD breaks even after 1887 ~ 645 prediction calls of 4096 points each, the testing setups follow a much lesser quantity: NL≤500N_L \leq 500 and NH≤100N_H \leq 100. The practicality of this speedup remains unclear. Additionally, in practical scenarios such as active learning, each iteration would cost ~65s to retrain the neural posterior compared with 1.8s of the co-kriging model, which is a disadvantage of the proposed method.

Although experimental results show a lower prediction and coverage errors than co-kriging, the improvement seems incremental (3.22% NRMSE reduction overall).


[1] Park, Sehyun, et al. "Knowledge Distillation of Uncertainty using Deep Latent Factor Model." Advances in Neural Information Processing Systems 38 (2026): 85673-85714.

[2] Markou, Stratis, et al. "Practical conditional neural processes via tractable dependent predictions." arXiv preprint arXiv:2203.08775 (2022).

Questions for authors

See weaknesses.

Formatting issues: currently all experimental result figures are clustered at the last two pages, making it difficult to read and relate to corresponding sections in the paper.

4 comments
AuthorsOpenReview ↗

Response to Reviewer DUzA

Dear Reviewer,

Thank you for your careful review and for recognizing the paper's presentation, sampling efficiency, and uncertainty estimates. We address your concerns below.

W1. Novelty and relation to prior work. We agree that covariance distillation itself is not new, and we do not claim it as a general contribution. Markou et al. develop conditional neural processes with tractable dependent predictions and exact maximum-likelihood training. Park et al. use a deep latent-factor model and EM to represent deep-ensemble predictive means and covariances. CA-MFPD addresses a different setting: under scarce high-fidelity (HF) data, it transfers a calibrated residual multi-fidelity Gaussian process (GP) posterior to a neural student using posterior-guided sampling and batchwise joint Gaussian KL, while observed HF responses anchor the student's mean.

To test the role of joint covariance transfer directly, we added a matched joint-versus-marginal Gaussian KL comparison over six benchmarks, three HF budgets, and five paired seeds. The comparison controls for the teacher, data splits, calibration, student, and optimization settings.

Held-out metricJoint KL lower (of 18 pairs)
Off-diagonal correlation RMSE18/18
Relative covariance Frobenius error15/18
Batch-mean variance error15/18

We also added a reconstructed perforated-plate stress-field experiment. Across ten LF training subsets, CA-MFPD obtains an NRMSE of 0.219169±0.0120300.219169\pm0.012030, compared with 0.25850.2585 for co-kriging.

W2. Training cost and practical use. We agree that CA-MFPD's one-time training cost is higher. The measured medians and repeated-batch break-even points are:

MethodTraining (s)Inference (ms / 4,096 points)Break-even batches vs. CA-MFPD
CA-MFPD64.973.742--
Co-kriging1.77937.2291,887
NARGP1.91540.0211,738
MFNN9.15690.225645

The revised Introduction identifies uncertainty propagation and dense spatial stress-field queries as workloads that may require many prediction batches after training. The break-even analysis concerns repeated prediction with a trained model and excludes retraining after each new HF observation, as in some active-learning loops.

W3. Magnitude of the predictive improvement. We agree that the 3.22% overall NRMSE reduction is modest and describe it accordingly. The mean NRMSE is 0.055162 for CA-MFPD and 0.056998 for co-kriging. We assess the method across several predictive dimensions: the median NLL is 0.507 versus 1.133, mean 90% coverage is 0.932 versus 0.858, and mean absolute coverage error is 0.101 versus 0.123. The held-out covariance comparison provides direct evidence for the cross-input dependence claim, separately from these pointwise metrics.

Q1. Figure placement. We revised the layout so the main learning-curve figure appears with the Results section, and the nonlinear-baseline comparison appears alongside its discussion. The break-even and ablation figures remain in the Appendix next to their corresponding analyses. Experimental figures are therefore distributed across the main results and relevant Appendix sections rather than collected only at the end.

Thank you for these comments. They helped us clarify the method's contribution, intended workload, and empirical evidence.

Dear Reviewer,

Thank you for your comments on the contribution and experimental evaluation of our paper. We have added an explanation of how multi-fidelity posterior transfer can be used in a cross-input engineering task.

We evaluate near-hole stress screening on SimuStruct perforated plates. For each geometry, we fit the models using a coarse-mesh low-fidelity stress field and six high-fidelity stress measurements. We then draw 4,000 samples from the student's full joint predictive distribution over near-hole locations to estimate the probability that the maximum near-hole von Mises stress exceeds 585.31 MPa.

We compute the class-balanced Brier score by first calculating the mean Brier score separately for exceedance and non-exceedance cases, and then averaging the two class-specific scores with equal weight. For each plate, we first average the predicted probabilities across five seeds. The results are as follows:

Number of plate geometriesExceedance casesJoint KLMarginal KL
102/100.04720.0941
95/90.15400.1699
10028/1000.23000.2655

Among the 100 additional geometries, joint KL detects 16 of the 28 exceedance cases, compared with 10 for marginal KL.

This experiment illustrates one use of multi-fidelity posterior transfer in spatial screening: using a small number of high-fidelity measurements and a low-fidelity stress field to predict the joint stress distribution across multiple near-hole locations, and then estimate the probability of a spatial maximum exceeding the threshold.

Thank you again for your valuable comments.

Dear Reviewer,

Thank you again for your great work on reviewing this paper submission to ICLR 2027.

The authors have now submitted their rebuttal. The open reviewer-author discussion phase will end soon, on November 18 at 11:59 PM AoE.

Please read the authors' responses, determine whether any further clarification is needed, and respond to the rebuttal at your earliest convenience. Please be sure to acknowledge the authors' responses and update your review accordingly.

Best regards, Your AC

Dear Reviewer,

Thank you again for your careful review. We have revised the paper to define our contribution more precisely as multi-fidelity posterior transfer under scarce high-fidelity data. We agree that covariance distillation itself is an established idea. For ease of review, we have marked the revised passages in red in the updated manuscript.

To evaluate the role of joint transfer, we added a matched joint-versus-marginal KL comparison that directly measures held-out cross-input dependence. We also added the SimuStruct screening study, where joint KL has lower class-balanced Brier scores in all three geometry cohorts. These experiments address the cross-input use of the method.

We have clarified the intended efficiency setting as repeated prediction with fixed training data, and acknowledge that frequent retraining in an active-learning loop remains costly.

We would appreciate hearing whether these revisions address your main concerns. If they do, we would be grateful if you would reconsider your assessment. We would be happy to clarify any remaining issue.

Sincerely, The Authors

Reviewer_r2ZLOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CA-MFPD distills a residual multi-fidelity GP into a neural predictor with a low-rank-plus-diagonal covariance. It combines joint Gaussian KL supervision, scalar uncertainty correction from high-fidelity leave-one-out residuals, reliability-weighted input sampling, and a high-fidelity mean anchor. An optional HF-only GP modifies training and prediction on Branin and Nonlinear-Sin. The analysis establishes finite-batch relationships between Gaussian KL, covariance approximation, and linear predictive queries. The evaluation covers ten benchmarks, seven HF budgets, and 20 seeds, with additional nonlinear baselines and a smaller ablation study

Critical strengths

CA-MFPD studies a useful problem of replacing a multi-fidelity Gaussian process with a faster neural predictor while retaining its uncertainty information. The evaluation across ten benchmarks and 20 seeds is a strength, and averaging nested data budgets before statistical testing is a sensible choice. The paper also reports cases where the method or individual components perform worse, which helps readers understand its limitations.

Critical weaknesses

The largest gap is that the experiments do not directly test the paper’s main contribution: learning how predictions at different inputs depend on one another. Prediction accuracy and uncertainty at individual inputs cannot establish this. The authors should compare the joint training loss against a loss that matches each input’s mean and variance separately, keeping the remaining setup fixed. They should then measure the learned covariance or test uncertainty for a quantity that combines several predictions. The existing mean-only ablation does not distinguish learning individual variances from learning relationships between inputs.

The optional high-fidelity GP safeguard also needs a clearer explanation and its own ablation. It is used on two benchmarks that contribute much of the aggregate improvement, but neither benchmark appears in the ablation study. Results with and without the safeguard would clarify its role. The final combined predictor’s covariance between different inputs also needs to be specified, since the paper currently gives only its mean and marginal variance. Architecture settings, loss weights, and comparable timing procedures should be provided so that the results can be reproduced.

A few minor reporting inconsistencies require correction. Appendix C.1 states that three benchmarks have no positive MFNN crossing, but Table 8 shows only one dash. Table 2’s printed win percentages sum to 102.9%. The authors should specify tie handling or correct the entries. Appendix C.1 states that three benchmarks have no positive MFNN crossing, but Table 8 shows only one dash.

Questions for authors

The defining covariance contribution is neither directly evaluated nor isolated experimentally. Proposition 4.1 establishes that marginal matching cannot identify off-diagonal covariance, but Figure 4 is an analytic example rather than an evaluation of the trained student. The experiments report no explicitly identified held-out joint KL, off-diagonal covariance error, or covariance-dependent predictive query (sec 6; Tables 1–9). NLL is not defined precisely enough to establish that it evaluates joint predictions. I don’t think the ablations resolve this gap. Appendix D describes mean-matching controls and a mean-only variant that removes joint KL. Neither is the marginal Gaussian KL objective of Eq. 10) under otherwise matched conditions. Removing joint KL changes both mean weighting and variance supervision, so Table 9 cannot attribute improvements specifically to cross-input dependence. I think a matched marginal-KL control and direct held-out dependence evaluation are really needed to substantiate the paper’s principal methodological distinction.

Overall, the evidence does not yet establish a substantial advantage attributable to the proposed integration’s distinguishing feature. While the theory derives generic properties of Gaussian KL (Section 4; Appendix A), I think the experiments principally establish accuracy and uncertainty-score behavior without directly measuring learned dependence. The missing matched marginal-KL comparison therefore limits both the novelty and significance assessment. A useful covariance-transfer result would materially strengthen the contribution beyond a competitive collection of mean-prediction and calibration components.

3 comments
AuthorsOpenReview ↗

Response to Reviewer r2ZL

Dear Reviewer,

Thank you for your careful review. We agree that the original experiments did not directly establish whether the student learned cross-input dependence. We have added controlled comparisons and clarified the safeguard and reporting details. Our point-by-point responses follow.

W1. Direct evidence of cross-input dependence. We added a matched comparison of joint and marginal Gaussian KL on six benchmarks, three nested HF budgets per benchmark, and five paired seeds. The variants share the teacher, data splits, calibration, student architecture and rank, initialization, Sobol augmentation points, loss weights, optimizer, training budget, and HF anchors. Marginal KL matches the mean and marginal variance at each input but omits cross-input covariance.

MetricTypeJoint KL lower (interval width: narrower)
NRMSEPointwise prediction9/18
NLLPointwise prediction12/18
CRPSPointwise prediction9/18
90% coverage errorPointwise prediction7/18
90% interval width (narrower)Pointwise prediction13/18
Relative covariance Frobenius errorCross-input dependence15/18
Off-diagonal correlation RMSECross-input dependence18/18
Batch-mean variance errorCross-input dependence15/18

The batch-mean metric evaluates predictive variance for an equal-weight average over a held-out input batch. The table separates pointwise predictive performance from direct cross-input dependence measures.

W2. HF-only GP safeguard. We added an on/off ablation on Branin and Nonlinear-Sin, with five seeds and three nested HF budgets per task. Paired variants share the data, test set, teacher, and student training settings. Results are averaged over budgets within each seed. On Branin, enabling the safeguard changes NRMSE from 0.0786±0.02190.0786\pm0.0219 to 0.0759±0.01740.0759\pm0.0174 and NLL from 4.761±0.1074.761\pm0.107 to 4.737±0.1474.737\pm0.147. On Nonlinear-Sin, NRMSE changes from 0.3151±0.10110.3151\pm0.1011 to 0.3035±0.10430.3035\pm0.1043 and NLL from 1.929±2.8201.929\pm2.820 to 1.684±2.8511.684\pm2.851. Training and inference costs increase modestly. Coverage changes from 0.922 to 0.917 on Branin and from 0.880 to 0.869 on Nonlinear-Sin.

W3. Reporting inconsistencies. The win percentages in Table 2 sum to 102.9% because every method tied for the lowest NRMSE in a cell is counted as a winner. We have stated this rule in the caption. We also corrected Appendix C.1: only Park-1 has no positive MFNN break-even point, consistent with the dash in Table 8.

Q1. Matched control and metric interpretation. The comparison in W1 changes only the distillation loss. We evaluate cross-input dependence directly through held-out covariance and correlation errors and a covariance-dependent batch-mean variance query. NLL averages univariate Gaussian negative log predictive densities over test inputs; it is a pointwise marginal score, not a joint batch likelihood, so we do not use it as evidence of learned cross-input covariance.

Q2. Covariance of the safeguarded predictor. For a query batch XX, the final covariance is KF(X)=a2KS(X)+(1−a)2KG(X)+a(1−a)Δμ(X)Δμ(X)⊤K_F(X)=a^2K_S(X)+(1-a)^2K_G(X)+a(1-a)\Delta\mu(X)\Delta\mu(X)^\top, where Δμ(X)=μS(X)−μG(X)\Delta\mu(X)=\mu_S(X)-\mu_G(X). Here KSK_S and KGK_G are the student and HF-only GP batch covariances. The pointwise variance formula in the paper gives the diagonal of KFK_F.

Q3. Relation to Gaussian Distillation. We do not claim covariance distillation itself as a new general idea. Park et al. use a deep latent-factor model and EM to represent the predictive mean and covariance of a deep ensemble. We address multi-fidelity regression with scarce HF data: posterior-guided data augmentation constructs training batches, an HF-LOO-calibrated residual multi-fidelity GP supplies the joint teacher posterior, and observed HF responses anchor the student mean. The contribution is this posterior-transfer method for the scarce-HF multi-fidelity setting. The matched comparison tests covariance transfer directly. We also added a perforated-plate stress-field experiment for sparse-HF spatial prediction; over ten LF training subsets, CA-MFPD obtains NRMSE 0.219169±0.0120300.219169\pm0.012030, versus 0.25850.2585 for co-kriging.

Thank you for your suggestions. They helped us evaluate the paper's central claim more directly.

Thank you for adding the requested matched joint-versus-marginal comparison and for doing direct covariance tests. The win percentage consistency makes sense. Thanks for clarifying. I would suggest to try making it clearer in the paper text that the precise contribution (as mentioned above) is your posterior-transfer method for the scarce-HF multi-fidelity setting instead of the broader idea of covariance distillation.

These address my major concerns, and I have increased my score.

Thank you for your constructive suggestion and for raising your score. We agree that the paper’s contribution should be stated more precisely. Thank you again for your careful review.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.