acceptodds
Under review as a conference paper at ICLR 2027

Generalized Matheron Variational Implicit Processes

Abstract

Implicit-process priors specify distributions over functions through sample-forward mechanisms such as Bayesian neural networks and stochastic simulators, but their function-space densities are typically unavailable. We introduce Generalized Matheron Variational Implicit Processes (GMVIP), a pathwise variational family for posterior inference with such priors. For Gaussian-process priors, GMVIP recovers the standard inducing-variable variational GP construction; for general implicit priors, its empirical covariance construction preserves the prior mean and covariance in the population limit. GMVIP constructs posterior samples by drawing a function from the prior and applying a correction anchored at a set of inducing inputs. The effect of this correction away from the inducing inputs is determined directly from prior samples, allowing the posterior to retain the structure and variability of the original implicit process. The (surrogate) prior and variational posterior use the same pathwise construction and differ only in the distribution of whitened inducing coefficients, yielding a tractable coefficient-space Kullback-Leibler divergence. Experiments on regression, classification, and forecasting with simulator-defined and retrieval-conditioned empirical trajectory priors show that GMVIP is broadly competitive with existing methods.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Implicit Processes, Variational Inference, Function-Space Inference

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_L6q1OpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes Generalized Matheron Variational Implicit Processes (GMVIP), a function-space variational inference method for implicit-process priors. The method combines prior function samples with Matheron-style corrections at inducing points, using empirical covariance estimates to construct posterior samples. It introduces Gaussian variational coefficients to obtain a tractable KL divergence under a surrogate prior. The authors show that the method recovers standard sparse variational Gaussian processes for Gaussian priors and preserves the first two moments for general implicit priors under suitable conditions. Experiments are conducted on regression, classification, and forecasting tasks, with competitive results against several existing methods.

Critical strengths
  1. The paper is generally readable, and the proposed method is relatively straightforward to understand.
  2. The authors provide theoretical analysis and experiments across regression, classification, and forecasting tasks. Some results, particularly on UCI regression and Lotka–Volterra, are promising.
Critical weaknesses
  1. Limited novelty. This is my main concern. The method largely combines existing ideas from Matheron sampling (Wilson et al., 2020), inducing-variable variational inference, and empirical covariance estimation. These components are already well established. It is unclear what the fundamental methodological contribution is beyond combining them.

  2. Unclear motivation and weak logical connection. The paper discusses the finite-basis limitation of VIP and the adversarial density estimation used by SIP, but these are two different problems. The authors do not clearly explain why a Matheron-style construction is the appropriate solution to both. The motivation feels somewhat disconnected from the proposed method, and the paper does not sufficiently justify why this particular combination is necessary.

  3. Limited theoretical contribution. Most theoretical results appear to follow from standard properties of Matheron sampling, Gaussian-process conditioning, and KL divergence. More importantly, for non-Gaussian implicit processes, the proposed construction does not generally preserve the original prior distribution, only its first two moments under suitable conditions. Therefore, it is unclear how faithfully the resulting variational posterior approximates the posterior under the original implicit prior.

  4. Experimental improvements are not sufficiently convincing. Although GMVIP performs well on some UCI datasets, its advantages are much less clear on larger benchmarks. On YearPredictionMSD, it is worse than FBNN, while the improvement on Airline is relatively small. On electricity forecasting, a simple Empirical Gaussian baseline achieves better RMSE, and the classification improvements are often marginal. Increasing the basis size of VIP/FTIP also substantially narrows the forecasting gap. Overall, the results do not demonstrate a consistent or substantial advantage.

  5. Fairness and computational cost concerns. GMVIP requires additional prior samples, empirical covariance estimation, and Matheron corrections, making it more expensive than several baselines. The comparisons are not consistently matched in computational budget. In addition, the electricity experiment optimizes observation noise for GMVIP but keeps it fixed for the Empirical Gaussian baseline, making the NLL comparison less convincing.

Questions for authors
  1. What is the specific advantage of using the Matheron covariance correction over other inducing-point interpolation operators? Since the tractable coefficient-space KL mainly comes from the shared generative mapping, is Matheron's rule really essential to the proposed framework?

  2. The proposed method only preserves the first two moments of a general non-Gaussian implicit prior. How can the authors justify that this is sufficient for accurate posterior inference, particularly when the original prior contains multimodal distributions or strong nonlinear dependencies?

  3. In the Lotka–Volterra experiment, the GMVIP surrogate prior has an ODE residual of approximately 0.31, compared with around 10⁻⁶ for the original prior. Doesn't this suggest that the proposed construction substantially changes the original process structure? How should this be reconciled with the motivation of preserving prior characteristics?

  4. In Appendix D.5, increasing the basis size substantially improves VIP and FTIP. For example, FTIP reaches an RMSE of 0.33 compared with 0.32 for GMVIP. Are these differences statistically significant? Would GMVIP still perform better under matched computational budgets?

  5. The authors jointly optimize the prior parameters and variational posterior in several experiments. How much of the reported improvement comes from the proposed posterior construction rather than prior adaptation? More importantly, how should the resulting data-dependent surrogate prior be interpreted from a Bayesian inference perspective?

  6. Given that GMVIP requires additional prior sampling and covariance estimation, could the authors provide a clearer analysis of its accuracy–efficiency trade-off? On the larger regression and classification benchmarks, the improvements often appear rather small. What practical advantages justify the additional computational cost?

Reviewer_v5NEOpenReview ↗
Rating
1(−1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes GMVIP, a function-space variational inference (FSVI) framework designed to achieve tractability while maintaining high fidelity to the original implicit prior.

Unlike conventional surrogate priors, which are constructed as linear combinations of functions sampled from the original prior, the authors introduce a novel pathwise surrogate prior based on Matheron's rule. By adding a residual path sampled from the true prior to a finite-dimensional representation based on inducing variables, their formulation preserves the infinite-dimensional variability of the original prior outside the inducing points, mitigating the representational collapse observed in previous methods.

The proposed method is evaluated on a wide variety of datasets, including synthetic and real-world regression benchmarks, ODE simulations, and time-series trajectories, and is compared against a large number of baseline methods.

Critical strengths
  1. Well-motivated and theoretically grounded approach:

The authors identify the limited expressiveness of surrogate priors constructed as linear combinations of sampled functions and address this limitation using an approach based on Matheron's rule to preserve the variability of the implicit prior processes. Furthermore, as demonstrated in Propositions 1–4, the theoretical formulation is mathematically sound, yielding a tractable coefficient-space KL regularizer while preserving the prior mean and covariance under the stated conditions.

  1. Applicability to diverse priors:

The proposed method demonstrates its effectiveness across various types of implicit priors, ranging from Bayesian neural network priors to priors constructed from empirical time-series trajectories.

Critical weaknesses

Mismatch between central claims and experimental design

A central strength of the proposed method is its high-fidelity surrogate prior, which aims to preserve the structural properties and variability of the original implicit prior. However, based on the current experimental results, it is difficult to isolate the factors responsible for the method's improved performance, as direct validation of this central claim is either missing from the main text or confounded by other factors. I suggest restructuring the experimental section to address the following interconnected issues:

  1. Insufficient emphasis on direct validation of the core contribution:

The surrogate-prior fidelity analysis (Appendix D.1) provides direct empirical evidence that the proposed method approximates the original prior more faithfully than VIP. This evidence is central to evaluating the effectiveness of the method and should be highlighted in the main text. To make room for this, I suggest condensing the extensive comparisons against a broad range of baselines (e.g., MAP, MFVI, and FBNN), which are less directly related to the core contribution.

  1. Lack of a fundamental ablation study:

To validate the paper's central claim, it is essential to disentangle the effects of the surrogate-prior construction (VIP-style vs. GMVIP-style) from those of the flexibility of the coefficient posterior (Gaussian vs. flow-based). For instance, in the Lotka–Volterra experiment (line 325, Fig. 2), FTIP appears to struggle to capture the dynamics. This raises the possibility that its weaker performance is attributable to the surrogate-prior construction rather than the expressiveness of the variational posterior. However, the current comparisons do not adequately isolate these two factors.

Questions for authors
  1. Ablation study:

To isolate the sources of the proposed method's advantages, particularly the relative contributions of the surrogate-prior construction and posterior expressiveness, could the authors provide an ablation study combining the GMVIP surrogate prior with a flow-based posterior (i.e., a GMVIP + Flow variant)?

  1. Relationship between GMVIP inducing-only and VIP:

In Appendix D.4, the GMVIP inducing-only model omits the residual path rg(x)r_g(x), making it appear similar to the VIP surrogate prior. Could the authors clarify the relationship between these two formulations and explain their differences?

Reviewer_RXknOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2(+1)
Presentation
1
Soundness
2(+1)
Read review
Paper summary

This paper introduces Generalized Matheron Variational Implicit Processes (GMVIP), a posterior inference framework designed for implicit-process priors (such as Bayesian neural networks and stochastic simulators) where function-space densities are unavailable and computing the process Kullback-Leibler (KL) divergence is intractable. Instead of restricting posterior draws to linear combinations of a pre-sampled, fixed functional basis as in Variational Implicit Processes (VIP), GMVIP generalizes Matheron's rule to implicit processes: it draws a fresh prior path, computes residuals against target values at a set of inducing inputs, and propagates this correction across the domain using an empirical covariance operator estimated from a prior sample bank. Because the surrogate prior and variational posterior share the same deterministic pathwise mapping, the intractable function-space divergence reduces exactly to a tractable KL divergence between finite-dimensional Gaussian distributions over inducing coefficients.

Critical strengths
  1. By incorporating Matheron's update into implicit process modeling, the method anchors posterior samples at inducing points while retaining the underlying residual prior path. This overcomes the expressiveness bottleneck of VIP, which confines all posterior realizations to the linear span of a small, fixed set of pre-sampled functions.
  2. The pathwise formulation resolves the numerical instability and variance collapse observed in earlier methods. In VIP, limited basis coverage can lead to near-zero predictive variance in under-represented input regions, causing severe numerical divergence in negative log-likelihood (NLL) evaluations. Retaining an explicit background prior path provides a baseline level of variance that prevents such collapse.
  3. On the Lotka–Volterra dynamical system forecasting benchmark, drawing entire ODE trajectory solutions as base paths enables the model to preserve physical oscillatory coupling over extrapolation horizons. The resulting ODE residual (0.31) is substantially lower than those of VIP (0.65) and FTIP (0.62).
Critical weaknesses
  1. The presentation and readability require improvement, placing an unnecessary cognitive burden on the reader. The introduction moves rather abruptly into the intractability of function-space KL divergences without first establishing the underlying practical objective, leaving the actual application setting unclear at the outset. In addition, the paper relies on several non-standard compound terms, such as "pathwise variational family," "whitened inducing coefficients," and "cardinal right-inverse." These terms are uncommon in the literature and are introduced without intuitive definitions or pedagogical context, making it harder to connect the prose to the underlying computations. I encourage the authors to adopt standard terminology where possible or provide clearer explanations upon their first appearance.
  2. The reliance on a linear additive correction presents conceptual and practical limitations when combined with highly non-linear implicit priors. A primary motivation for implicit processes is to preserve non-Gaussian, non-linear structures generated by deep networks or simulators. However, the empirical Matheron operator in Equation (11) remains a linear update based on second-order sample covariance. For dynamical systems with hard boundary constraints (such as non-negativity in population sizes) or non-linear phase shifts, additive linear adjustments may not fully respect the underlying physical state evolution. The authors should discuss the domain of validity and the structural limitations of this linear formulation more thoroughly in the main text.
  3. The attribution of performance gains between prior adaptation and the core variational operator remains ambiguous in the main text. Section 5 notes that the UCI regression experiments employ an empirical-Bayes setup that jointly tunes the prior network parameters. However, the ablation study in Appendix D.4 shows that when the prior parameters are kept frozen, the performance margins of GMVIP over baseline methods narrow considerably on several datasets, such as Boston and Energy. While these results are reported in the appendix, the main text presents the tunable-prior numbers without sufficiently qualifying how much of the observed improvement stems from prior adaptation rather than the Matheron update itself.
Questions for authors
  1. The introduction motivates the method primarily through the computational intractability of function-space KL divergences, which makes the narrative fairly abstract. My reading is that the core practical objective is to calibrate a black-box simulator or complex prior against noisy observations while providing reliable uncertainty bounds, but standard variational inference stalls because the lack of an explicit density prevents evaluating the KL regularizer that safeguards against overfitting. Does this interpretation align with the authors' original motivation? If so, I suggest clarifying this concrete task pipeline early in the introduction to make the practical goals more transparent.
  2. Could the authors clarify the precise mathematical interpretation of the phrase "pathwise variational family" as used in the paper, and explain how it fundamentally differs from standard push-forward variational distributions or latent-variable generative models over function spaces?
  3. In Equation (11), the correction term is added linearly to the sampled prior path. If the underlying implicit simulator has natural non-negativity constraints (such as physical concentrations or population counts), how does this linear adjustment ensure that the resulting trajectory does not produce negative values? Are any projection steps or constraint-handling techniques applied in the implementation?
  4. Regarding the frozen-prior ablation in Appendix D.4, could the authors provide further insight into the degree to which GMVIP depends on prior parameter adaptation? In a setting where the simulator is strictly non-differentiable and its internal parameters cannot be tuned, what relative level of performance can be expected?
  5. The identity in Proposition 2 depends on the exact interpolation condition ΨZ(Z)=I\Psi_Z(Z) = I. In practice, inverting the empirical covariance matrix K^(Z,Z)\hat{K}(Z, Z) almost always requires adding a diagonal jitter term λI\lambda I for numerical stability. Does the addition of this jitter affect the strict equality in Equation (13) in the actual implementation?

Comments on the paper

1 comment

Dear AC and reviewers,

We would like to kindly follow up on our rebuttal responses. As the discussion period is approaching its end, we would greatly appreciate it if the reviewers could take a moment to read our responses and any additional clarifications provided.

We hope that our answers have adequately addressed the concerns raised during the review process. If the reviewers find our responses satisfactory, we would be grateful if they could consider updating their assessments and, where appropriate, increasing their scores.

We remain available to address any further questions or concerns before the discussion period closes.

Thank you very much for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.