- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Jointly training the encoder and decoder in amortized Bayesian inference can lead to poorly informative summaries. The paper proposes to augment the standard inference objective with a direct loss on the summary to predict point estimates of the posterior parameter's functionals.
Critical strengths
- The basic statistical motivation is sound.
- The paper is generally easy to follow.
Critical weaknesses
-
The novelty of the work seems limited. In Section 3.1, the paper offers a plausible explanation, but does not prove, theoretically or empirically, the claimed saddle or slow escape for deep encoder-decoder posterior estimators.
-
The only methodological contribution seems to be training the encoder and decoder jointly with a point estimate guidance loss (Section 3.3). Such losses already appear in the approximate Bayesian inference literature: Fearnhead and Prangle learn posterior-mean summaries and then perform ABC; Jiang et al. learn neural regression summaries and then perform ABC. Jointly training encoders and decoders is again standard in the neural posterior estimation literature, e.g. BayesFlow. Could the authors clarify what exactly is new?
-
A closer published paper (Schosser et al) first pre-trains the summary network with parameter regression and then finetunes with joint NPE loss. Is training jointly from the start better than this?
-
Another principal limitation is the empirical diagnostic interpretation. Low CCA can reflect nonlinear encoding rather than lost information, as Appendix A.2 explicitly acknowledges. Conversely, high CCA with posterior means does not establish posterior sufficiency. Direct mean supervision further explicitly encourage a representation that scores well under this diagnostic. Therefore, claims that guidance “fully resolves” the representation gap in the non-Gaussian examples (p. 6), or that CCA confirms insufficient summaries as the cause (Conclusion, p. 9), are not fully supported.
References
Fearnhead, P., & Prangle, D. (2010). Constructing summary statistics for approximate bayesian computation: Semi-automatic abc. arXiv preprint arXiv:1004.1112.
Jiang, B., Wu, T. Y., Zheng, C., & Wong, W. H. (2017). Learning summary statistic for approximate Bayesian computation via deep neural network. Statistica Sinica, 1595-1618.
Schosser, B., Heneka, C., & Plehn, T. (2024). Optimal, fast, and robust inference of reionization-era cosmology with the 21cmPIE-INN. arXiv preprint arXiv:2401.04174.
Questions for authors
See weaknesses.
4 comments
Response to W3:
We thank the reviewer for this reference, which we will discuss in the revision. Schosser et al. correspond to one configuration of our guidance (K = 1, MSE loss, S = D), applied as a separate pre-training stage before joint training. Notably, they report that without pretraining their jointly trained networks did not converge. This is consistent with the failure mode we characterize and provides independent evidence for it.
As to the delineation with our contribution, we would like to point out that their diagnosis is limited to “the combination of both networks did not converge when jointly trained, probably due to the large difference in number of network parameters between the two networks”.
Our analysis offers a mechanistic account of the failure they observed:
Summaries that are uninformative early in training receive suppressed gradients. The network can therefore settle in a partial posterior with no first-order signal to escape it. Pretraining toward the posterior mean ensures initialization far from these, which explains why their fix works.
Further, their fix requires multi-stage training (pre-training a posterior mean estimator, then individual training of the inference network, then joint training) which introduces time and complexity costs, as well as restricts the number of learnt summaries to the number of parameters, which is not generally advisable since sufficiency often requires higher summary dimensionality.
Our generalization with joint training with either direct guidance in summary space or indirect guidance via a prediction head that forms an inference network ensemble together with the jointly trained original inference network, gives tangible improvements:
- It removes the need for hyperparameter tuning across different training stages and
- allows for summary dimensions that are free to learn to be maximally informative and distinct from point estimates.
We expect joint training to at least match the staged approach while needing a single training run and allowing for improvements due to . We will add an experiment with sequential training to the first case study (specifically to the multivariate Gaussian experiments), closely mimicking Schosser et al., to corroborate this. We will post results as they come in.
Response to W4:
We thank the reviewer for this careful point. We agree and acknowledge that CCA measures linear recoverability and a low value can reflect non-linear encoding, and a high value does not imply posterior sufficiency. Our motivation is the failure mode described in Section 3.1, where the inference network erroneously falls back to the prior in certain parameter directions due to uninformative summaries, reflected by poor recovery of the corresponding posterior means. Thus, we use CCA to assess whether the learned summaries retain linearly recoverable information about all posterior-mean directions, with the minimum canonical correlation identifying the weakest direction. In practice, accurate recovery across all parameter directions is a highly desirable property for well-concentrated, unimodal posteriors, which are abundant in practice. Further, the utility of CCA has strong precedent in the representation learning literature for assessing linear recoverability of information [1, 2, 3].
Our conclusions do not rest on CCA alone. We diagnose summary failure on the unguided networks, where no supervision favours CCA, and there a low coincides with poor RMSE and contraction for the same parameters. We then measure the benefit of guidance with RMSE, posterior contraction, calibration error, and expected log probability. These are computed from posterior samples of the inference network, which the auxiliary losses do not optimize directly. Furthermore, ensemble guidance never supervises the summaries directly, yet it still improves these metrics.
We agree that the two sentences on p.6 and p.9 were strongly worded and we thank the reviewer for pointing them out. We will revise p.6 to state that guidance narrows the representation gap as measured by these metrics against the reference posteriors. For conclusion on p.9, we will reframe it to state that our proposed CCA diagnostic results are consistent with insufficient summaries.
We hope our responses have addressed the reviewer’s concerns and clarified the scope of our contribution. We are happy to provide any further details or clarification.
Thank you!
References
[1] https://arxiv.org/abs/1706.05806
Dear Reviewer,
as promised, we have added a proof of the partial posterior being a stationary and a saddle point. We agree it complements the more general argument (Sec. 3.1) and the empirical evidence (Sec. 5) well.
Thus, in the most recent revision, we have (marked in blue)
- ... modified section 3.1 to explain the general phenomenon why summary gradients can be suppressed in the first order.
- ... added a section to the appendix where we are able to precisely characterize the nature of the stationary point, proving it to be a saddle point, after stating more explicit conditions.
- ... added a toy example, with a linear summary network and a Gaussian inference network. It serves as an toy model amenable to analytic treatment, exemplifying the existence of saddle points at partial posterior solutions.
We are also working on further experimental evidence, including a comparison with sequential training stages, as Schosser et al are using, on inference tasks in experiment 1 and 2. We will add the results when the full suite of experiments has concluded, but in the meantime can confirm our earlier prediction that sequential training is not improving on joint training. At equal training step budget and on identical training data, we consistently see joint training to match or exceed the accuracy reached with sequential training. The only exception is the toy normal task, 5.1 (a), where the posterior means are exactly sufficient, such that the lack of flexibility to learn complementary summaries happens to become an apt inductive bias for this specific example. In all other settings, sequential training reached lower or equal performance, even though it always had a final stage of joint training that Schosser et al. proposed to overcome issues due to sequential training.
We hope our responses, including the ones in previous comments, have addressed your concerns. As the author-reviewer discussion period ends soon, we would be grateful if you could let us know whether you have any remaining questions that we can further clarify. If our response has resolved your concerns, we would kindly ask you to consider reflecting this in your evaluation.
Thank you!
I thank the authors for their detailed response. The additions satisfactorily address my concerns, and I am happy to increase my score. All the best :)
We thank the reviewer for the constructive feedback and for confirming the statistical motivation of our work. We believe that the reviewer has three major concerns: i) novelty relative to existing literature, ii) evidence for the training dynamics in sec. 3.1, and iii) interpretation of the CCA diagnostic. We agree that point-estimate regression and joint training are each well established and neither is our contribution. Our contribution is threefold. First, we identify representation gaps as a systematic failure mode of standard end-to-end ABI, where the summary network gets stuck even though the architecture can represent the needed information. Second, we propose a diagnostic that separates summary failures from inference failures. Third, we introduce a general framework of auxiliary guidance losses applied jointly throughout training. These losses are simple, widely applicable, and cheap to compute compared to existing approaches. Together, these contributions address critical weaknesses of ABI in practice, as shown by our extensive experiments.
Response to W1:
We have presented a theoretical description of the suppression of gradients towards escaping the partial posterior in 3.1. Indeed, throughout our experiments, we see clear evidence of that slow escape. Figure 2 shows this most clearly. In each of three runs with different seeds, training first plateaus, then escapes. The escape appears as a step down in the representation gap (measured by posterior accuracy) and, at the same time, a step up in summary informativeness (measured by CCA). The timing of the step differs between seeds. The auxiliary guidance gives further evidence for the correctness of our theoretical claim. Guidance raises summary informativeness directly, and the representation gap closes as a result.
We agree that a formal proof under which assumptions these partial posterior solutions are saddle points of the loss would be a valuable addition. Indeed, we should be able to provide such a proof in the camera ready version, under certain assumptions on the summary architecture.
Here is a sketch of that proof:
If the inference network returns the partial posterior p(θ(i)∣s(i)),p(θ(u)), it ignores s(u), so the gradient carries no information about θ(u) (Eq. 5). If moreover s(i) and s(u) are produced by separable weights, this point is stationary. The loss is flat in the weights producing s(u), so the Hessian has the form (0 C⊤CA) with A⪰0, and any nonzero coupling C makes it a saddle, while with C=0 it could also be a local minimum.
We can also add a linear-Gaussian example, where the negative curvature scales with how strongly the data inform θ(u).
Please let us know if such additions to the paper would be sufficient in your view, and clarify if and where you would find the theoretical or empirical evidence still lacking. On novelty, we have responded to it in our opening paragraph and provide further clarification in the W2 and W3 responses below.
Response to W2:
Fearnhead & Prangle and Jiang et al. use regression to construct the summary itself, which is then fixed and passed to ABC. Their summary is therefore limited to D point estimates, which are generally not sufficient for the full posterior, and it is never adapted to the inference task. In our framework, point-estimate losses act only as an auxiliary signal during end-to-end NPE training. The summaries remain optimized for the posterior, they can have more dimensions than there are parameters (S ≥ (K+1)D, Sec. 3.3), and in the ensemble variant they are not directly constrained at all. Furthermore, their regression summaries correspond to a limiting case of our framework, with a single point estimate (K = 1), no free summary dimensions (S = D), and the summary network trained on the auxiliary loss alone.
Joint training is indeed standard and theoretically sound. Our contribution is to show that, in practice, it can fail systematically through representation gaps, and to provide (i) a diagnostic that detects such failures and separates them from inference failures, and (ii) a general solution that does not require extensive hyperparameter tuning. We also clearly state our novelty in the introduction.
We continue the remaining responses in the comment below.