CA-MFPD: Covariance-Aware Multi-Fidelity Posterior Distillation under Scarce High-Fidelity Data

Multi-fidelity regression uses cheap, biased low-fidelity data to reduce the need for expensive high-fidelity evaluations. Gaussian process (GP) models provide joint predictive distributions, but repeated GP inference can be costly. We propose covariance-aware multi-fidelity posterior distillation (\method), which trains a neural surrogate model to reproduce the teacher's predictive distribution. We use a residual multi-fidelity GP to predict high-fidelity outputs at sampled inputs, then adjust predictive uncertainty using leave-one-out (LOO) errors on high-fidelity data. During training, a joint Gaussian KL loss brings the student's predictive mean and covariance closer to the teacher's. The student represents its covariance as a low-rank term plus a diagonal term. The off-diagonal entries represent correlations between inputs. We also compute the student's mean error at observed high-fidelity points and include it in the training objective. Our analysis shows that joint KL compares correlations between inputs, which pointwise comparison omits. We bound batch KL error in terms of mean and covariance approximation errors, and show that a lower expected batch KL gives a tighter bound on average pairwise covariance error. Numerical results show that CA-MFPD has lower overall prediction and coverage errors and faster repeated inference than standard co-kriging, with competitive results against nonlinear GP and neural multi-fidelity baselines.