acceptodds
Under review as a conference paper at ICLR 2027

Mitigating Representation Gaps in Amortized Bayesian Inference with Auxiliary Supervision

Abstract

Casting Bayesian inference as a neural network optimization problem targeting an amortized posterior is attractive, as it extends to otherwise intractable statistical models and offers near instantaneous inference for new datasets after prepaying the training cost. Although theory guarantees faithfulness under ideal convergence, practical amortized inference still requires iterating over architectures and optimization choices and ultimately “satisficing” under finite simulation, compute, and time budgets. Even the best-performing solution may thus retain avoidable representation gaps that typically require problem-specific fixes. Here, we propose a generic alternative which improves training dynamics with auxiliary guidance losses applied to internal representations. Specifically, we show how such guidance leads to faster convergence when training data is abundant and to better performance when it is scarce. We formalize representation gaps as getting stuck in a local optimum at the information bottleneck between the parts of the network tasked with feature learning and those tasked with conditional distribution learning, and offer a generic diagnostic to separate summary failures from inference failures. Finally, we demonstrate that auxiliary supervision improves convergence speed and accuracy on a range of challenging real-world inference problems.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
amortized-bayesian-inference, simulation-based-inference, training-dynamics, information-bottleneck, point-estimation

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_Mz2ROpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Jointly training the encoder and decoder in amortized Bayesian inference can lead to poorly informative summaries. The paper proposes to augment the standard inference objective with a direct loss on the summary to predict point estimates of the posterior parameter's functionals.

Critical strengths
  • The basic statistical motivation is sound.
  • The paper is generally easy to follow.
Critical weaknesses
  • The novelty of the work seems limited. In Section 3.1, the paper offers a plausible explanation, but does not prove, theoretically or empirically, the claimed saddle or slow escape for deep encoder-decoder posterior estimators.

  • The only methodological contribution seems to be training the encoder and decoder jointly with a point estimate guidance loss (Section 3.3). Such losses already appear in the approximate Bayesian inference literature: Fearnhead and Prangle learn posterior-mean summaries and then perform ABC; Jiang et al. learn neural regression summaries and then perform ABC. Jointly training encoders and decoders is again standard in the neural posterior estimation literature, e.g. BayesFlow. Could the authors clarify what exactly is new?

  • A closer published paper (Schosser et al) first pre-trains the summary network with parameter regression and then finetunes with joint NPE loss. Is training jointly from the start better than this?

  • Another principal limitation is the empirical diagnostic interpretation. Low CCA can reflect nonlinear encoding rather than lost information, as Appendix A.2 explicitly acknowledges. Conversely, high CCA with posterior means does not establish posterior sufficiency. Direct mean supervision further explicitly encourage a representation that scores well under this diagnostic. Therefore, claims that guidance “fully resolves” the representation gap in the non-Gaussian examples (p. 6), or that CCA confirms insufficient summaries as the cause (Conclusion, p. 9), are not fully supported.

References

Fearnhead, P., & Prangle, D. (2010). Constructing summary statistics for approximate bayesian computation: Semi-automatic abc. arXiv preprint arXiv:1004.1112.

Jiang, B., Wu, T. Y., Zheng, C., & Wong, W. H. (2017). Learning summary statistic for approximate Bayesian computation via deep neural network. Statistica Sinica, 1595-1618.

Schosser, B., Heneka, C., & Plehn, T. (2024). Optimal, fast, and robust inference of reionization-era cosmology with the 21cmPIE-INN. arXiv preprint arXiv:2401.04174.

Questions for authors

See weaknesses.

4 comments

We thank the reviewer for the constructive feedback and for confirming the statistical motivation of our work. We believe that the reviewer has three major concerns: i) novelty relative to existing literature, ii) evidence for the training dynamics in sec. 3.1, and iii) interpretation of the CCA diagnostic. We agree that point-estimate regression and joint training are each well established and neither is our contribution. Our contribution is threefold. First, we identify representation gaps as a systematic failure mode of standard end-to-end ABI, where the summary network gets stuck even though the architecture can represent the needed information. Second, we propose a diagnostic that separates summary failures from inference failures. Third, we introduce a general framework of auxiliary guidance losses applied jointly throughout training. These losses are simple, widely applicable, and cheap to compute compared to existing approaches. Together, these contributions address critical weaknesses of ABI in practice, as shown by our extensive experiments.

Response to W1:

We have presented a theoretical description of the suppression of gradients towards escaping the partial posterior in 3.1. Indeed, throughout our experiments, we see clear evidence of that slow escape. Figure 2 shows this most clearly. In each of three runs with different seeds, training first plateaus, then escapes. The escape appears as a step down in the representation gap (measured by posterior accuracy) and, at the same time, a step up in summary informativeness (measured by CCA). The timing of the step differs between seeds. The auxiliary guidance gives further evidence for the correctness of our theoretical claim. Guidance raises summary informativeness directly, and the representation gap closes as a result.

We agree that a formal proof under which assumptions these partial posterior solutions are saddle points of the loss would be a valuable addition. Indeed, we should be able to provide such a proof in the camera ready version, under certain assumptions on the summary architecture.

Here is a sketch of that proof:

If the inference network returns the partial posterior p(θ(i)∣s(i)),p(θ(u))p(\theta^{(i)}\mid s^{(i)})\\,p(\theta^{(u)}), it ignores s(u)s^{(u)}, so the gradient carries no information about θ(u)\theta^{(u)} (Eq. 5). If moreover s(i)s^{(i)} and s(u)s^{(u)} are produced by separable weights, this point is stationary. The loss is flat in the weights producing s(u)s^{(u)}, so the Hessian has the form (0C C⊤A)\begin{pmatrix}0 & C \\\ C^\top & A\end{pmatrix} with A⪰0A \succeq 0, and any nonzero coupling CC makes it a saddle, while with C=0C=0 it could also be a local minimum.

We can also add a linear-Gaussian example, where the negative curvature scales with how strongly the data inform θ(u)\theta^{(u)}.

Please let us know if such additions to the paper would be sufficient in your view, and clarify if and where you would find the theoretical or empirical evidence still lacking. On novelty, we have responded to it in our opening paragraph and provide further clarification in the W2 and W3 responses below.

Response to W2:

Fearnhead & Prangle and Jiang et al. use regression to construct the summary itself, which is then fixed and passed to ABC. Their summary is therefore limited to D point estimates, which are generally not sufficient for the full posterior, and it is never adapted to the inference task. In our framework, point-estimate losses act only as an auxiliary signal during end-to-end NPE training. The summaries remain optimized for the posterior, they can have more dimensions than there are parameters (S ≥ (K+1)D, Sec. 3.3), and in the ensemble variant they are not directly constrained at all. Furthermore, their regression summaries correspond to a limiting case of our framework, with a single point estimate (K = 1), no free summary dimensions (S = D), and the summary network trained on the auxiliary loss alone.

Joint training is indeed standard and theoretically sound. Our contribution is to show that, in practice, it can fail systematically through representation gaps, and to provide (i) a diagnostic that detects such failures and separates them from inference failures, and (ii) a general solution that does not require extensive hyperparameter tuning. We also clearly state our novelty in the introduction.

We continue the remaining responses in the comment below.

Response to W3:

We thank the reviewer for this reference, which we will discuss in the revision. Schosser et al. correspond to one configuration of our guidance (K = 1, MSE loss, S = D), applied as a separate pre-training stage before joint training. Notably, they report that without pretraining their jointly trained networks did not converge. This is consistent with the failure mode we characterize and provides independent evidence for it.

As to the delineation with our contribution, we would like to point out that their diagnosis is limited to “the combination of both networks did not converge when jointly trained, probably due to the large difference in number of network parameters between the two networks”.

Our analysis offers a mechanistic account of the failure they observed:

Summaries that are uninformative early in training receive suppressed gradients. The network can therefore settle in a partial posterior with no first-order signal to escape it. Pretraining toward the posterior mean ensures initialization far from these, which explains why their fix works.

Further, their fix requires multi-stage training (pre-training a posterior mean estimator, then individual training of the inference network, then joint training) which introduces time and complexity costs, as well as restricts the number of learnt summaries to the number of parameters, which is not generally advisable since sufficiency often requires higher summary dimensionality.

Our generalization with joint training with either direct guidance in summary space or indirect guidance via a prediction head that forms an inference network ensemble together with the jointly trained original inference network, gives tangible improvements:

  1. It removes the need for hyperparameter tuning across different training stages and
  2. allows for summary dimensions that are free to learn to be maximally informative and distinct from point estimates.

We expect joint training to at least match the staged approach while needing a single training run and allowing for improvements due to S>DS>D. We will add an experiment with sequential training to the first case study (specifically to the multivariate Gaussian experiments), closely mimicking Schosser et al., to corroborate this. We will post results as they come in.

Response to W4:

We thank the reviewer for this careful point. We agree and acknowledge that CCA measures linear recoverability and a low value can reflect non-linear encoding, and a high value does not imply posterior sufficiency. Our motivation is the failure mode described in Section 3.1, where the inference network erroneously falls back to the prior in certain parameter directions due to uninformative summaries, reflected by poor recovery of the corresponding posterior means. Thus, we use CCA to assess whether the learned summaries retain linearly recoverable information about all posterior-mean directions, with the minimum canonical correlation identifying the weakest direction. In practice, accurate recovery across all parameter directions is a highly desirable property for well-concentrated, unimodal posteriors, which are abundant in practice. Further, the utility of CCA has strong precedent in the representation learning literature for assessing linear recoverability of information [1, 2, 3].

Our conclusions do not rest on CCA alone. We diagnose summary failure on the unguided networks, where no supervision favours CCA, and there a low ρmin\rho_{\text{min}} coincides with poor RMSE and contraction for the same parameters. We then measure the benefit of guidance with RMSE, posterior contraction, calibration error, and expected log probability. These are computed from posterior samples of the inference network, which the auxiliary losses do not optimize directly. Furthermore, ensemble guidance never supervises the summaries directly, yet it still improves these metrics.

We agree that the two sentences on p.6 and p.9 were strongly worded and we thank the reviewer for pointing them out. We will revise p.6 to state that guidance narrows the representation gap as measured by these metrics against the reference posteriors. For conclusion on p.9, we will reframe it to state that our proposed CCA diagnostic results are consistent with insufficient summaries.

We hope our responses have addressed the reviewer’s concerns and clarified the scope of our contribution. We are happy to provide any further details or clarification.

Thank you!

References

[1] https://arxiv.org/abs/1706.05806

[2] https://arxiv.org/abs/1806.05759

[3] https://arxiv.org/abs/1610.01644

Dear Reviewer,

as promised, we have added a proof of the partial posterior being a stationary and a saddle point. We agree it complements the more general argument (Sec. 3.1) and the empirical evidence (Sec. 5) well.

Thus, in the most recent revision, we have (marked in blue)

  1. ... modified section 3.1 to explain the general phenomenon why summary gradients can be suppressed in the first order.
  2. ... added a section to the appendix where we are able to precisely characterize the nature of the stationary point, proving it to be a saddle point, after stating more explicit conditions.
  3. ... added a toy example, with a linear summary network and a Gaussian inference network. It serves as an toy model amenable to analytic treatment, exemplifying the existence of saddle points at partial posterior solutions.

We are also working on further experimental evidence, including a comparison with sequential training stages, as Schosser et al are using, on inference tasks in experiment 1 and 2. We will add the results when the full suite of experiments has concluded, but in the meantime can confirm our earlier prediction that sequential training is not improving on joint training. At equal training step budget and on identical training data, we consistently see joint training to match or exceed the accuracy reached with sequential training. The only exception is the toy normal task, 5.1 (a), where the posterior means are exactly sufficient, such that the lack of flexibility to learn complementary summaries happens to become an apt inductive bias for this specific example. In all other settings, sequential training reached lower or equal performance, even though it always had a final stage of joint training that Schosser et al. proposed to overcome issues due to sequential training.


We hope our responses, including the ones in previous comments, have addressed your concerns. As the author-reviewer discussion period ends soon, we would be grateful if you could let us know whether you have any remaining questions that we can further clarify. If our response has resolved your concerns, we would kindly ask you to consider reflecting this in your evaluation.

Thank you!

I thank the authors for their detailed response. The additions satisfactorily address my concerns, and I am happy to increase my score. All the best :)

Reviewer_ZnuJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors decompose the role of an inference network into two steps: 1.) mapping to a summary of the data that serves as a sufficient statistic of the approximate posterior, and 2.) mapping from the summary statistic to the approximate posterior itself. Since imperfect summary statistics limit the available training signal for optimising the mapping to the approximate posterior, and likewise, an imperfect mapping to the approximate posterior limits the available training signal for optimising the mapping to summary statistics, regular SBI/amortised inference training pipelines allow for local optima/saddlepoints to disrupt optimisation. To remedy this, the authors propose to use an additional term in the objective function that directly promotes sufficiency of the summary statistic, meaning the part of the network responsible for obtaining the summary representation is less sensitive to an imperfect latter part of the network. The authors also introduce a metric that is designed to measure the degree of sufficiency of the summary representation. The authors evaluate their proposed method across a wide variety of inference network architectures and experimental settings, and the results are impressive.

Critical strengths

S1 In general, the writing is exceptionally clear. The problem that the authors aim to solve is explained well and the rest of the paper is very well motivated.

S2 Section 3.1 is especially good$\textemdash$outstanding in fact. It provides a very good intuition for how representation gaps might arise and the intuition is well-supported by equation 5.

S3 I am very impressed by the diversity of experimental settings considered in this paper. Rarely have I been as convinced of the efficacy of a proposed method as when reading this paper. The fact that the proposed method does not lead to performance improvements in the mouse experiment should not be viewed as a limitation of the method; it is further evidence that the proposed method directly targets representation gaps, a pathology that is demonstrated not to be present in that experiment through the fact that the unguided method matches the point estimator in mean parameter recovery.

Critical weaknesses

W1: CCA. The authors consider summary and parameter vectors that live respectively in RS\mathbb{R}^S and RD\mathbb{R}^D. But somehow the authors compute covariance matrices from variations within individual vectors. I do not see how this is possible; perhaps the covariance matrices are obtained by considering a set of such vectors and the variation across individual vector samples? Regardless, the authors need to correct / clarify this section, as this diagnostic is one of the paper's main contributions.

W2: Method Presentation. Despite a very high quality of writing for most of the paper, I find the section that introduces the authors' core contribution$\textemdash$Section 3.3$\textemdash$to be difficult to understand. Please can the authors clarify the exact form that each ℓk\ell_k takes and explain how a loss is computed between a summary vector and parameter vector$\textemdash$vectors that live in different spaces. Do these loss terms relate to CCA and ρR\rho_R that were introduced in the previous section, or are they completely unrelated? Furthermore, it is not clear to me why the authors consider an enumeration over KK (or K+1K+1) sets of summary vectors for each x,θ\boldsymbol{x}, \boldsymbol{\theta} pair$\textemdash$from Section 2 it is my understanding that a single summary statistic carries all the information needed to parameterise the full posterior distribution, so why would KK different ones be needed? The exposition of the core contribution should not leave readers in such a confused state, so it is very important that the authors work to make this subsection more clear.


If the authors address these two clarity-related problems, I would be happy to vote in favour of acceptance as I believe the rest of the paper is very strong. But currently, as a paper whose core contribution remains incomprehensible (to me at least), I cannot vote in favour of acceptance.

Questions for authors

Further Requests

R1 The introduction's presentation of amortised Bayesian inference is too SBI-centric. There are plenty of ABI settings in which there is no simulation at all, such as the neural process setting or generative modelling scenarios involving e.g. VAEs. The introduction should be edited to respect the broader scope of ABI over SBI.

R2 In Figure 1, there is a method in the plots which is not included in the legend or description of the experiment. It is denoted by an upward-pointing triangle △\triangle. Please can the authors clarify which method this is?

R3 For experiment 2, only one "guided" method is displayed. For this, did the authors apply the ensemble or the direct summary guidance method? And is there any particular reason why only one approach was used/reported in this experiment? Answers to these questions should be available in the paper.

R4 It would be helpful for readers if the authors added suitable signage to their figures (↑\uparrow and ↓\downarrow) to indicate whether a metric corresponds to higher-is-better or lower-is-better. This is especially important for Figures 3, 4, and 5, whose plots correspond to metrics with different desired directions. This signage can be added in Figure captions, plot axis labels, or directly overlaid onto plots themselves.

Editorial Suggestions

E1 On lines 81 and 82, the sentence would be clearer if em-dashes were added in the following way: "learns maximally informative$\textemdash$and thus sufficient (Halmos & Savage, 1949)\textemdash\textemdashsummary statistics".

E2 On lines 140-146, the difference between Cov⁡(⋅)\operatorname{Cov}(\cdot) and Σ⋅\boldsymbol{\Sigma}_\cdot is not clear to me. Is there a difference? If not, just one of these symbols should be used.

E3 On line 226, the citation for the flow-matching paper should be parenthetical rather than textual.

E4 On line 258, "permits" should be swapped out for "admits".

6 comments

We thank the reviewer for their valuable feedback and positive thoughts on our paper. In the revised version, we will address the points concerning clarity as well as the additional requests.

Response to W1:

Thank you for pointing out the need for extra clarification. Indeed, the covariance matrices are not computed from variations within individual vectors, but are estimated across a set of parameter-summary pairs obtained from multiple datasets. Each dataset provides one summary vector and one corresponding target vector (either the ground-truth parameters or a reference posterior point estimate). CCA is then applied to this collection of paired vectors, with covariances estimated across dataset instances. We have revised Section 3.2 to explicitly introduce this collection and to clarify how the covariance matrices are computed, as follows:

We consider a set of MM pairs of ground-truth parameters and summary vectors {(θ(m),s(m))}m=1M\lbrace (\boldsymbol{\theta}^{(m)},\mathbf{s}^{(m)})\rbrace_{m=1}^M, where s(m)=sψ(x(m))∈RS\mathbf{s}^{(m)}=s_\psi (\mathbf{x}^{(m)}) \in \mathbb{R}^S and θ(m)∈RD\boldsymbol{\theta}^{(m)} \in \mathbb{R}^D. The covariance matrices are estimated across these MM pairs. CCA then separately whitens the summary and parameter spaces using their respective covariance matrices, and computes the singular values of their whitened cross-covariance matrix (Equation 6).

Response to W2:

We thank the reviewer for this comment and agree that Section 3.3 needs to be clearer. We will revise it accordingly.

Relationship of the loss to CCA: The loss terms are not related to CCA. The CCA from Section 3.2 is only a diagnostic for detecting a possible representation gap and is not used in the loss.

Choice of KK and form of ℓk\ell_k: KK is the number of point estimates the summary network is guided toward. Besides the posterior mean, one can also use, e.g., the variance, median or several other quantiles. Each ℓk,k∈1,…,K\ell_k, k \in \\{1,\dots, K\\}, is the scoring rule that matches the kk-th point estimate. For example, ℓk\ell_k is the squared error loss for the posterior mean and the pinball loss for a quantile. We will state the exact form of ℓk\ell_k in the revised section. In the case studies, we focussed only on the posterior mean (hence K=1K = 1), which may have caused the confusion. Does this answer your questions? If not please let us know and we are happy to clarify further.

Response to Further requests:

R1 We agree that the introduction is too SBI-centric. We will broaden it to cover ABI settings without simulators, such as neural processes and VAEs, and add the relevant references to the related work.

R2 Figure 1 has two legends, one after the first two rows and one at the bottom, and the first is easy to overlook. We will merge them into a single, clearly visible legend that includes the method marked by the upward-pointing triangle.

R3 We thank the reviewer for this observation. The guided method shown in Experiment 2 is the summary-guided method. We omitted the ensemble-guided results from the main text to keep the figure readable. We agree, however, that this choice was not explained in the paper and may cause confusion. In the revised version, we will state explicitly in the main text which guidance method is shown and also add a figure with the ensemble-guided results to the appendix.

R4 We agree. We will add ↑\uparrow and ↓\downarrow indicators to the axis labels, in particular in Figures 3–5, and explain them in the captions.

Response to Editorial Suggestions:

We sincerely thank the reviewer for reading our paper so carefully and for these helpful editorial suggestions. We will incorporate all of them in the revised version.

I thank the authors for their clear and informative rebuttal.

W1 That makes much more sense to me now, and the proposed Section 3.2 revision text looks much better to me. The authors have addressed this concern of mine fully.

W2 This is also much clearer to me now, and I hope that the revised version of Section 3.3 will exhibit this new clarity. My current understanding is as follows$\textemdash$please could the authors could confirm whether it is correct?

For K=1K=1 and a guidance loss chosen as the squared error loss (which is the authors' default experimental setting), the guidance loss encourages the first DD dimensions of the summary vector to be as close as possible to the posterior mean of the parameters, and then the remaining dimensions are free to form whatever representation so as to further minimise the Bayes risk. And if various point estimate guidance terms are used (K>1K>1), the optimal summary vector would take on a block-wise structure in which each DD dimensional snippet of the vector becomes close to whatever posterior point estimate of the parameters that snippet's guidance loss corresponds, with the final DD dimensional snippet being free to form whatever representation would further minimise the Bayes risk.

R1-4 These are resolved by the authors' proposed revisions.

Thank you for the prompt and positive response. Regarding W2: your understanding is correct for the summary-guided approach, in which the guidance is applied directly to the summary vector. One clarification: the number of unguided dimensions does not have to equal DD. It is simply the number of summary dimensions that remain after the guided ones.

Reviewer_h1SpOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The paper tackles an interesting problem that asks if a neural network can learn an approximation of a bayesian posterior distribution while reducing the representational gap. For that, the authors propose to add auxiliary loss functions in their objective function.

Therefore, they propose to distinguish between a summary failure and an inference failure. The first refers to the case where the encoder discards important information and is shadowed by the dominant signals, whereas the inference failure refers to the observation where the encoder retains relevant information but lacks in learning the correct distributions.

This leads to the integration of auxiliary loss objectives such that these additional training signal helps to learn useful representation of the posterior distributions.

Critical strengths

S1: Separation of summary and inference failure

The analysis of the representation gaps via distinguishing between summary learning and the posterior estimation seems appealing. So it gives evidence on the “why” an estimator might perform bad apart from the inference architecture.

S2: Lightweight integration

In the same spirit, the authors provide a lightweight approach by just integrating the loss objectives into any existing pipeline. So, it augments the training objective with the point estimation losses to force the summary network to preserve the relevant information.

Critical weaknesses

W1: Budget

In Section 5, the authors state that every estimator is compared in a setup, where the auxiliary losses are the expressive part rather than the architecture or capacity. In equation 7, the authors introduce their joint loss by leveraging the loss indicators for each of the k-th point estimate. In an slight adaptation of this setup, the authors propose in Equation 8 to additionally include MLPs to learn the representation of a summary s. In the evaluation part, if we use the setup described in equation 8, we increase the capacity by construction. How do the authors assure a equal budget in this specific setup? Seems like the ensemble guided setup indeed introduces additional parameters, therefore increasing model capacity and expressiveness, that are not given for the unguided baseline.

W2: summary network

In the experimental setup of the experiment 3 - eye movements, the authors explicitly state that they leverage a set transformer as the summary network. What were the observations in sticking to the setup used apparently in all the other networks such that here, a change was needed? There is the impression in the room, that the effectiveness itself indeed depends on the summary architecture and its implementation rather than the auxiliary supervision. Can the authors give some hints and facts about the improvements across different configurations? Also in Appendix A.6, the authors state for experiment 2 the following: “In this experiment we exceptionally use BayesFlow in version 2.0.11, instead of 2.0.12, which includes a modification of the set transformer that is used as the summary network. The improvements within release 2.0.12 proved to be sufficient to largely fix the representation gap with default configurations”. Therefore, are the improvement attributed to the auxiliary losses integration or other unrelated improvements in the architectural design. So, it remains unclear to me if we can really attribute the lift to the adapted objective function.

W3: Experimental setup for experiment 1

The authors describe the standard normal prior and produce 10 iid observations from a “different location-scale” distribution corresponding to different sufficient statistics. They then describe the normal distribution with a fixed scale, a uniform distribution, and a student-t-distribution with fixed scale. Apparently, with the restriction of fixed scale distributions, we are reducing the complexity of the task at-hand. May I ask if the authors also conduct experiments under a relaxation of this constraint, where we have scales also sampled from a pre-defined interval. What are the OOD capabilities in this regard for the proposed approach? It would be a harder task to infer jointly the scale information with the location parameters, I’d assume. Because then, the summary network needs to track multiple characteristics of the likelihood.

W4: No HPO / Competitor Analysis

Another point then thinking about W2 again. What I’d also be curious about is if the proposed optimization is the reason for improvement or can we reach the same performance by conducting a more thorough hyperparameter search. We see in the experimental section the convergence behavior and also training duration but what about alternative optimization strategies with the same resource constraints. So any comparison to other optimization strategies are left out of the evaluation section.

W5: CCA

In appendix A.2, the authors state: “a small ρmin may reflect a non-linear dependence of μ”. Now, can the authors elaborate why CCA fits their excpectations to identify summary failures and inference failures. One can have a high correlation between the posterior mean and the point estimates, but why is the assumption that the summary network captures all the information to reciver the full posterior distribution. Is this a consequence of the fixed scale setting the authors suggest? And if a small p_min might reflect a non-linear dependence, then the approach itself is only able to identify linear relationships which should be made transparent or at least a more in-depth clarification in this point.

Questions for authors
  • Q1: Can the authors clarify how they make sure that the experiments are conducted under a fair computational budget, specifically for the ensemble-guided approach where additional MLP are leveraged and making therfore the architecture more expressive by design. (W1)
  • Q2: Did the authors conduct additional experiment that provide insights into the sensitivity towards the architecture used as summary network? From the current setup, it seems like a substantially dependence on the architectural design of this backbone rather than on the additional auxiliary losses. (W2)
  • Q3: What about relaxations in the experimental setup of 1, where we are not constraint the setup to fixed scales, but relax them in a way such that the summary network is forced to retain multiple aspects of the distribution. Did the authors conduct experiments in this more general setting? (W3)
  • Q4: Did the authors conduct experiments with a more thorough hyperparameter sweep to justify that the lift indeed comes from the auxiliary losses rather than decisions referring to the architectural design or the hyperparameter settings being leveraged? What about a comparison to other optimization techniques? (W4)
  • Q5: Can the authors elaborate a bit more on the aspect they described in Appendix A2 which makes the usage of CCA as a metric that has been used throughout the experiments questionable? (W5)
5 comments

We thank the reviewer for their valuable feedback. Several concerns seem to come from reading our contribution as an improvement in architecture, capacity, or competitor to optimization strategies, which it is not. Representation gaps are a training-dynamics failure: a local optimum at the bottleneck between the summary and inference networks, which occurs across architectures. Our auxiliary supervision is an architecture-agnostic addition to the training objective that overcomes it. Within each experiment, every estimator therefore uses the same summary network, inference network, optimizer, schedule and simulation budget, and the only difference is the training signal. For the same reason, auxiliary guidance is not an alternative optimization strategy. It complements any optimizer or hyperparameter choice and can be combined with them. With this in mind, we address the specific concerns below.

Response to W1 and Q1

The ensemble-guided variant does add point-estimation MLPs fkf_k, but these extra parameters do not increase the capacity or expressiveness of the posterior estimator. The MLPs are used only during training, where they contribute additional gradients to the shared summary network. Ensemble guidance applies the auxiliary losses to separate MLPs, so the summary network receives extra training signals but stays free to shape its representation that best serves posterior estimation. At inference time they are not utilized, and posterior samples come from the flow-based inference network alone, whose architecture, input, and summary dimension are identical to the unguided baseline.

Moreover, summary guidance adds no parameters at all, yet it improves results in Exp. 1–4. Finally, guidance helps only where a gap exists. In Exp. 5, three summary networks already achieve high CCA without guidance, and guidance brings no further improvement. A gain driven by added capacity would not be this selective.

Response to W2 and Q2

The choice of summary network is determined by the structure of the data. Exp. 1–3 have set-structured data, so we use set-based summary networks there: a set transformer in Exp. 1 and 2, and both a deep set and a set transformer in Exp. 3. Exp. 4 uses a CNN because its data are images. Exp. 5 has graph-structured data, so we compare four summary networks, two of them are graph-based. The set transformer in Exp. 3 is therefore for the inductive bias it provides and not a departure from other experiments.

On sensitivity to architecture, the paper already reports it. In Exp. 3 we pair both summary networks with three inference networks and report all six combinations across guidance types and weights (Figs. 8–11, App. A.7). Exp. 5 makes the same comparison across four summary networks. Across the experiments, guidance improves results over a range of summary architectures, which would be unlikely if the gains came from the architecture rather than the auxiliary supervision.

Regarding BayesFlow 2.0.11: In Exp. 2, the guided and unguided runs share the same version and the same network, so the version cannot explain the improvement. Modifications to the set transformer in v2.0.12 happened to remove the gap for this specific experiment. However, in Exp. 1 and 3, both of which use the v2.0.12 set transformer, unguided training still leaves a representation gap that guidance reduces. It shows that representation gap is a fragile training-dynamics effect that depends on small architectural details. Practitioners cannot know in advance which tweak will fix which task, whereas guidance is a generic fix that needs no such search.

Response to W3 and Q3

We would like to clarify that Exp. 1 already includes the varying scale setting. In the Uniform task, we infer both the center and the half-width of each of 16 blocks (D = 32), so the summary network must learn location and scale together. The tasks are also not easy as we go up to 64 dimensions with only 10 observations, and the Student-t distribution has no sufficient statistic smaller than the full data. Even at 100k training simulations, unguided training still shows a gap for Normal and Uniform whereas our approach clearly fixes it.

We keep Exp. 1 deliberately controlled as it has exact or near-exact reference posteriors and known sufficient statistics, so the gap can be precisely measured. In Exp. 2, the parameters that stay unrecovered without guidance are exactly the noise standard deviations σ\sigma and sτs_{\tau}. Also, guidance clearly shows improvements in more complex experiments (Exp. 2, 3, and 4).

Regarding OOD, the representation gap is an in-distribution training failure, so OOD behaviour is outside the scope of this work. As noted in Sec. 4, guidance does not change the inference target, and we make no OOD claims.

We continue the remaining responses in the comment below.

Response to W4 and Q4

We would like to clarify that auxiliary guidance is not an alternative optimization strategy, so it does not compete with optimizers or hyperparameter tuning. It is a general addition to the NPE training objective that works alongside any choice of optimizer, schedule, or architecture and can be combined with all of them. To isolate its effect, every estimator within an experiment shares the same optimizer, schedule, and simulation budget.

We do perform hyperparameter tuning in favour of the unguided baseline in Exp. 1. We grid-searched dropout for the unguided baseline and also tested longer training, and the gap continued to persist (Appendix A.5). Whereas, guidance itself needs no per-task tuning as its weight is set by normalizing the initial losses.

We agree that a task-specific change can sometimes close the gap, as BayesFlow 2.0.12 did for Exp. 2. However, such fixes cannot be known in advance and do not carry over to other tasks. Our contribution is a cheap, generic way to steer the summary network towards representations that are informative for the posterior, closing or substantially narrowing the gap without an expensive task-specific search.

Response to W5 and Q5

We thank the reviewer for raising this point. We agree and acknowledge that CCA measures linear relationships and does not, by itself, imply the sufficiency of learned summaries for recovering the full posterior. Our motivation is the failure mode described in Section 3.1, where the inference network erroneously falls back to the prior in certain parameter directions due to uninformative summaries, reflected by poor recovery of the corresponding posterior means. Thus, we use CCA to assess whether the learned summaries retain linearly recoverable information about all posterior-mean directions, with the minimum canonical correlation identifying the weakest direction. In practice, accurate recovery across all parameter directions is a highly desirable property for well-concentrated, unimodal posteriors, which are abundant in practice. Further, the utility of CCA has strong precedence in the representation learning literature for assessing linear recoverability of information [1, 2, 3].

We also agree that low canonical correlation does not necessarily imply information loss, as the posterior mean may be encoded nonlinearly. We thus interpret CCA as a diagnostic of linear recoverability, not as a general test of sufficiency. Accordingly, in our experiments, we complement it with parameter-specific RMSE, posterior contraction, calibration, and expected log probability to assess posterior approximation quality holistically. We have clarified these distinctions and limitations in Sections 3.2 and A.2.

We hope our responses have addressed the reviewer’s concerns and clarified the scope of our contribution. We are happy to provide any further details or clarification.

Thank you!

References

[1] https://arxiv.org/abs/1706.05806

[2] https://arxiv.org/abs/1806.05759

[3] https://arxiv.org/abs/1610.01644

Dear Reviewer,

We hope our response has addressed your concerns. As the author–reviewer discussion period ends soon, we would be grateful if you could let us know whether you have any remaining questions that we can further clarify. If our response has resolved your concerns, we would kindly ask you to consider reflecting this in your evaluation.

Thank you!

Reviewer_C8qUOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper address the problem of so-called representation gap in bayesian amortized inference. More specifically, they assume variational family qϕ(θ∣sψ(x))q_{\phi}(\theta| s_{\psi}(x)) is parametrized by two neural networks, a inference network qϕq_{\phi} and a summary network sψ.s_{\psi}. The two networks are jointly optimized in training. The presentation gap is characterized by the failure to learn the maximally informative summary statistics. The considered amortized inference framework is general which captures standard parametric amortized inference and I believe it would also apply to simulation based inference where one can amortize over multiple datasets (then s(x)s(x) would be something like deep set idea).

Their idea to narrow the representation gap is to add a penalty term (or loss term) that captures the correlation between summary vector (or their functions with these functions themselves learnable) and the parameter vectors. Through guidance from this additional penalty term, they are hoping to improve the training dynamics over the standard amortized bayesian inference, and escape these saddle points.f

Critical strengths

The paper is well written and explains the motivation behind the problem very well. They proposed an intuitive solution by adding an additional loss for guidance the training of amortized bayesian inference.

I personally like the method that it is so simple

the negative results in the mice example is also helpful

Critical weaknesses

I actually have some concern about the CCA diagnostic. CCA only applies for linear prediction or predictability. A sufficient summary can have low canonical correlation with a parameter or its posterior mean if the relevant relationship is nonlinear. The non-linear part is probably ok as the author introduces an additional layer of learnable function f_js. But overall CCA cannot distinguish summary failure from inference failure. I saw in the main text that it interprets low CCA as evidence of missing information. On the other hand, high CCA with posterior means establishes only that the summaries preserve first-moment information.

Another weakness in my opinion is that the paper lacks a formal account of local optima. The explanation is intuitive and I generally was excited when reading the paper especially starting section 3, for example Equation (5) correctly illustrates that ignoring summary coordinates can suppress gradients through those coordinates, which is very intuitive. But one should formally establish that the joint parameter configuration is stationary, a saddle point, or a local minimum? And how does the new training improve the convergence with a formal analysis? but the explanations are still heuristic. I would be more comfortable with the paper being published with a formal convergence analysis. Some things that can be easily shown include given the current guided framework, say gradients with respect to inference-network parameters need not vanish. also one can show shared summary-network parameters can modify multiple coordinates simultaneously and uninformative summary coordinates do not imply that the unresolved parameters have their prior distribution conditional on all remaining summaries (addressing what was brought up in section 3.1) .

Basically, can you focus on a a tractable example and provide an explicit stationary-point and curvature analysis? This is really one of the key things that's missing which undermine my enthusiasm as a reader.

The author claims The claim that minibatch noise masks escape directions. This seems makes sense, but can you test it or demonstrate it.

Questions for authors

see weakness above

3 comments

We thank the reviewer for the constructive feedback. We are glad that the reviewer finds the paper easy to follow and considers the simplicity of our method a strength.

We address the main concerns below.

W1. CCA diagnostic

CCA does indeed only measure linearly decodable information. A high canonical correlation with the posterior mean implies that first-moment information is preserved, not that the summaries are sufficient. Conversely, a low CCA does not strictly imply that the summaries are insufficient. We will highlight both points explicitly in Section 3.2 and A.2, where we interpret CCA as a diagnostic of linear recoverability, not as a general test of sufficiency. We will also sharpen the interpretation in the text.

Nevertheless, we still consider CCA an informative diagnostic.

Our motivation is the failure mode in Section 3.1 where the inference network falls back to the prior in certain parameter directions when the summaries are uninformative. We use CCA to assess whether the learned summaries retain linearly recoverable information about all posterior-mean directions. Accurate recovery is a highly desirable property for well-concentrated, unimodal posteriors, which are abundant in practice. The utility of CCA has a precedence in the representation learning literature for assessing linear recoverability of information [1,2,3].

As we state in Section 2.2, "learned summaries need not only be informative, but their representation needs to be usable by the inference network; the inference network itself is pushed to ignore not immediately exploitable summaries." A representation gap is not only about whether information exists in the summaries, but about whether the inference network can efficiently use it. Information encoded linearly is picked up more easily, whereas information encoded nonlinearly is harder for the inference network to learn from. A nonlinear summary encoding may still constitute a practical representation gap.

We do not rely on CCA as a sole diagnostic, and we complement it with parameter specific metrics as well as the expected joint log probability to assess posterior approximation error holistically. Across our case studies, low CCA coincides with poor performance: poorly recovered parameters, long training plateaus, and lower expected log probability. Improved CCA through guidance goes together with improved recovery and expected log probability. When CCA is already high without guidance (Exp. 5), guidance brings no improvement, as expected.

To avoid any confusion, we point out that guidance itself does not depend on CCA: CCA is merely a diagnostic. The guidance losses nudge the networks away from the partial posterior described in Section 3.1. We chose the posterior mean because it is often (weakly) sufficient, but the method does not rely on the posterior mean or any other chosen statistic being the most informative or sufficient. Further, guidance does not restrict summaries to first-moment information. In Section 3.3, we recommend S ≥ (K+1)D, so that at least D summary dimensions remain free from explicit guidance. These free dimensions allow the summary network to encode higher-moment information or anything else not captured by the guided directions. With ensemble guidance, the summary dimensions are entirely free; guidance acts only through the summary network shared by the inference networks in the ensemble.

W2. Lack of a more thorough analysis

We appreciate the reviewer’s interest in a more detailed analysis of Section 3.1. Currently, the section explains the mechanism through which the gradient signal vanishes, but we do not formally prove the resulting configuration to be a stationary point, saddle, or local minimum. We would like to improve the paper in this direction. We are currently preparing a revised Section 3.1 with a more precise description of the phenomenon, and an appendix with a formal analysis of the saddle point. We will provide an updated draft.

Regarding minibatch noise: We could test this empirically. For example in Experiment 2, we could measure the gradient signal through the uninformative summary directions and compare it with the minibatch gradient noise, with and without guidance, around the escape from the plateau.

Please let us know if such additions to the paper would be sufficient. We plan to add our results here later once we obtain them.

We thank the reviewer again for the helpful suggestions and are grateful for any additional feedback.

References:

[1] https://arxiv.org/abs/1706.05806

[2] https://arxiv.org/abs/1806.05759

[3] https://arxiv.org/abs/1610.01644

Thank you for suggesting a formal curvature analysis. We have added a proof of the partial posterior being a saddle point in precisely stated conditions. Indeed, we agree it complements the more general argument (Sec. 3.1) and the empirical evidence (Sec. 5) well.

In the most recent revision, we have (marked in blue)

  1. ... modified section 3.1 to explain the general phenomenon why summary gradients can be suppressed in the first order.
  2. ... added a section to the appendix where we are able to precisely characterize the nature of the stationary point, proving it to be a saddle point, after stating more explicit conditions.
  3. ... added a toy example, with a linear summary network and a Gaussian inference network. It serves as an toy model amenable to analytic treatment, exemplifying the existence of saddle points at partial posterior solutions.

Further, we are currently working on an additional experiment comparing the gradients on the uninformative summaries in escape direction (towards encoding the missing information) with the minibatch noise. We are focusing on experiment 2 where a plateau was seen both in CCA and in posterior quality (representation gap) in the two scale parameters σ\sigma and \sτ\s_\tau. There, we can measure that the signal-to-noise ratio is higher when auxiliary summary supervision is applied, compared to when it is not. Further, we can see that the the total pull towards informative summaries is not just smaller and noisier in the absence of auxiliary supervision, also the pull attributed to the inference network's loss grows when the auxiliary loss has moved summaries away from the partial posterior solution (meta stability).

We will add these experimental results to the manuscript later.

Dear Reviewer,

this comment is to notify you that the most recent revision includes the promised experiment demonstrating that the escape signal is small compared to the minibatch noise.

In stochastic training, the weak escape signal on the summary weights competes with minibatch noise from the dominant first term, which keeps trying to refine the uninformative summaries. In Appendix 3, we confirm this by measuring the escape signal and its variation across minibatches.

Quoting from the appendix A.3, see the PDF for the full description of how we measured the escape signal strength and its noise.

Figure 6 shows the escape signal relative to its minibatch noise, next to the error of the unresolved parameters σ\sigma and sτs_\tau. Gradient decent does indeed escape the partial posterior only slowly without auxiliary supervision (B&C, black), and we see that this is due to the escape signal being weak and noisy (A, black). Further, in the guided run, we see how the terminal loss on the inference network (A, red, dashed) starts out to be as irrelevant as in the unguided case (black), but since the auxiliary loss has pushed the summaries far enough away from the gradient-suppressing partial posterior solution, picks up signal strength off its own already after a few epochs.


Thank you for suggesting this direction of improvement! We believe together with the formal proof (see above and in the revision, A.2) our exposition of "representation gaps" has sharpened, beyond the more general argument and empirical evidence that we had initially put forward.

We hope our response has addressed your concerns. As the author–reviewer discussion period ends soon, we would be grateful if you could let us know whether you have any remaining questions that we can further clarify. If our response has resolved your concerns, we would kindly ask you to consider reflecting this in your evaluation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.