- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper proposes Generalized Matheron Variational Implicit Processes (GMVIP), a function-space variational inference method for implicit-process priors. The method combines prior function samples with Matheron-style corrections at inducing points, using empirical covariance estimates to construct posterior samples. It introduces Gaussian variational coefficients to obtain a tractable KL divergence under a surrogate prior. The authors show that the method recovers standard sparse variational Gaussian processes for Gaussian priors and preserves the first two moments for general implicit priors under suitable conditions. Experiments are conducted on regression, classification, and forecasting tasks, with competitive results against several existing methods.
Critical strengths
- The paper is generally readable, and the proposed method is relatively straightforward to understand.
- The authors provide theoretical analysis and experiments across regression, classification, and forecasting tasks. Some results, particularly on UCI regression and Lotka–Volterra, are promising.
Critical weaknesses
-
Limited novelty. This is my main concern. The method largely combines existing ideas from Matheron sampling (Wilson et al., 2020), inducing-variable variational inference, and empirical covariance estimation. These components are already well established. It is unclear what the fundamental methodological contribution is beyond combining them.
-
Unclear motivation and weak logical connection. The paper discusses the finite-basis limitation of VIP and the adversarial density estimation used by SIP, but these are two different problems. The authors do not clearly explain why a Matheron-style construction is the appropriate solution to both. The motivation feels somewhat disconnected from the proposed method, and the paper does not sufficiently justify why this particular combination is necessary.
-
Limited theoretical contribution. Most theoretical results appear to follow from standard properties of Matheron sampling, Gaussian-process conditioning, and KL divergence. More importantly, for non-Gaussian implicit processes, the proposed construction does not generally preserve the original prior distribution, only its first two moments under suitable conditions. Therefore, it is unclear how faithfully the resulting variational posterior approximates the posterior under the original implicit prior.
-
Experimental improvements are not sufficiently convincing. Although GMVIP performs well on some UCI datasets, its advantages are much less clear on larger benchmarks. On YearPredictionMSD, it is worse than FBNN, while the improvement on Airline is relatively small. On electricity forecasting, a simple Empirical Gaussian baseline achieves better RMSE, and the classification improvements are often marginal. Increasing the basis size of VIP/FTIP also substantially narrows the forecasting gap. Overall, the results do not demonstrate a consistent or substantial advantage.
-
Fairness and computational cost concerns. GMVIP requires additional prior samples, empirical covariance estimation, and Matheron corrections, making it more expensive than several baselines. The comparisons are not consistently matched in computational budget. In addition, the electricity experiment optimizes observation noise for GMVIP but keeps it fixed for the Empirical Gaussian baseline, making the NLL comparison less convincing.
Questions for authors
-
What is the specific advantage of using the Matheron covariance correction over other inducing-point interpolation operators? Since the tractable coefficient-space KL mainly comes from the shared generative mapping, is Matheron's rule really essential to the proposed framework?
-
The proposed method only preserves the first two moments of a general non-Gaussian implicit prior. How can the authors justify that this is sufficient for accurate posterior inference, particularly when the original prior contains multimodal distributions or strong nonlinear dependencies?
-
In the Lotka–Volterra experiment, the GMVIP surrogate prior has an ODE residual of approximately 0.31, compared with around 10⁻⁶ for the original prior. Doesn't this suggest that the proposed construction substantially changes the original process structure? How should this be reconciled with the motivation of preserving prior characteristics?
-
In Appendix D.5, increasing the basis size substantially improves VIP and FTIP. For example, FTIP reaches an RMSE of 0.33 compared with 0.32 for GMVIP. Are these differences statistically significant? Would GMVIP still perform better under matched computational budgets?
-
The authors jointly optimize the prior parameters and variational posterior in several experiments. How much of the reported improvement comes from the proposed posterior construction rather than prior adaptation? More importantly, how should the resulting data-dependent surrogate prior be interpreted from a Bayesian inference perspective?
-
Given that GMVIP requires additional prior sampling and covariance estimation, could the authors provide a clearer analysis of its accuracy–efficiency trade-off? On the larger regression and classification benchmarks, the improvements often appear rather small. What practical advantages justify the additional computational cost?
Dear AC and reviewers,
We would like to kindly follow up on our rebuttal responses. As the discussion period is approaching its end, we would greatly appreciate it if the reviewers could take a moment to read our responses and any additional clarifications provided.
We hope that our answers have adequately addressed the concerns raised during the review process. If the reviewers find our responses satisfactory, we would be grateful if they could consider updating their assessments and, where appropriate, increasing their scores.
We remain available to address any further questions or concerns before the discussion period closes.
Thank you very much for your time and consideration.