- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes amortized factor inference networks (AFIN), which encode priors and likelihoods as factors with known types and parameters. Through shared encoding, factor attention mechanisms, and a decoding network, it generates a Gaussian posterior approximation, which can be further refined using the SNIS method. Synthetic and real-data experiments show its accuracy and low inference costs.
Critical strengths
-
The authors utilized the structure of the posterior to propose a pre-trained network that covers multiple prior-likelihood combinations, which can be used to quickly producing usable approximations.
-
The architecture handles the product structure of the factors, allowing the same weights to accommodate different parameter settings, latent dimensions, and observation counts.
Critical weaknesses
-
Neural posterior estimation already offers flexible approximations through Gaussian mixtures [1] and normalizing flows [2,3]. AFIN instead uses a single Gaussian with a strictly diagonally dominant precision matrix. This restricts its flexibility. Although the authors suggest that self-normalized importance sampling (SNIS) can be used for correction, which is a common adjustment method in Monte Carlo sampling (Also, SNIS may affected by the curse of dimensionality). With SNIS, I feel more like AFIN provides a proposal distribution for posterior sampling.
-
The SNIS correction requires likelihood evaluation. However, many popular simulation-based inference methods, including neural posterior estimation, neural likelihood estimation, and neural ratio estimation, require only the ability to simulate from the likelihood [4].
[1]. Papamakarios and Murray. "Fast ε-free inference of simulation models with bayesian conditional density estimation", 2016.
[2]. Durkan, et al. "Neural spline flows", 2019.
[3]. Greenberg et al., "Automatic posterior transformation for likelihood-free inference", 2019
[4]. Lueckmann, et al. "Benchmarking simulation-based inference", 2021.
Questions for authors
-
In Fig. 2 and Fig. 3, AFIN achieves lower Mean L2 and Sliced W2, but other methods like NUTS and VI do not seem to yield good results even after convergence. Is this because the runtime was insufficient, or do NUTS and VI fail on this type of problem, or are they targeting the wrong distribution?
-
The training method (see lines 314-321) only on combinations of three priors and four likelihoods. If there are only a limited number of such combinations, how much additional benefit does the specialized factor architecture provide compared to a standard conditional neural posterior estimation that takes the same input information? If, in practical applications, the model has a complex likelihood (for example, Generalized linear model, but the likelihood is not among the types used during training), will the model still demonstrate excellent performance?
2 comments
Q2. Model diversity, conditional NPE, and unseen likelihoods.
A small factor library still defines a nontrivial inference family. With prior configurations, likelihood forms, and observations, there are unordered heterogeneous type-count configurations, where binom is the binomial coefficient. For example, allowing up to five likelihood forms per task gives 744,174,320 possible configurations at , , , , before varying numerical parameters. These are possible supported inputs. Our model class includes standard Bayesian GLMs and also permits different supported likelihood families within the same task. In our heterogeneous benchmark with and , eight scalar linear predictors leave at least eight latent directions unconstrained by the likelihood. With Student-t priors, the posterior retains polynomial tails along these directions.
The paper's ablation compares attention-based merging with sum pooling. Keeping node-and-pair representations, factor adapters, and the Gaussian decoder, Table 19 reports direct 0.1324 for sum pooling versus 0.0412 for AFIN under the same training simulator and schedule. The sum-pooling variant is smaller, so this comparison reflects both architecture and capacity.
New likelihood types can be added by training an adapter. The submitted Poisson extension evaluates 24 tasks and three seeds:
| New-type training | Direct ↓ | SNIS ↓ | Time (GPU-seconds) |
|---|---|---|---|
| Adapter only | 0.0591 | 0.0301 | 17,820 |
| Joint fine-tuning | 0.0458 | 0.0305 | 30,204 |
Training only the new adapter, with the shared network frozen, gives comparable SNIS accuracy with about 40% less GPU time than joint fine-tuning. This demonstrates an economical extension to a new likelihood; performance on other new families would need validation after adaptation.
References
[1] Papamakarios and Murray. Fast ε-free Inference of Simulation Models with Bayesian Conditional Density Estimation. NeurIPS, 2016. https://arxiv.org/abs/1605.06376
[2] Greenberg et al. Automatic Posterior Transformation for Likelihood-Free Inference. ICML, 2019. https://proceedings.mlr.press/v97/greenberg19a.html
[3] sbi v0.26.1; NPE_C implementation. https://github.com/sbi-dev/sbi
[4] Owen. Monte Carlo theory, methods and examples. Chapter 9, Theorem 9.2. https://artowen.su.domains/mc/Ch-var-is.pdf
[5] Revised Figure 4: SNIS comparisons on data-scarce and heterogeneous tasks. https://anonymous.4open.science/r/ICLR_AFIN-F43C/figures/figure4.pdf
[6] Khan and Rue. The Bayesian Learning Rule. JMLR, 2023. https://jmlr.org/papers/v24/22-0291.html
[7] Revised Figure 2: synthetic benchmarks. https://anonymous.4open.science/r/ICLR_AFIN-F43C/figures/figure2.pdf
[8] Revised Figure 3: UCI benchmarks. https://anonymous.4open.science/r/ICLR_AFIN-F43C/figures/figure3.pdf
Thank you for your review. We revised selected manuscript figures and added NPE and matched SNIS comparisons to clarify the accuracy, cost, and role of our Gaussian decoder.
Flow-based NPE is more expressive, but fitted accuracy and per-model training cost also matter. We compared NPE [1] and three-round SNPE-C [2] with neural spline flows using sbi v0.26.1 [3], 100,000 simulations per model setup, and one training seed. Each panel contains six tasks, with two per synthetic difficulty. These are direct predictions without SNIS; SW2 denotes sliced Wasserstein-2 distance.
AFIN has lower average direct error on both panels, in about 0.01 seconds versus 296–701 seconds per task. NPE/SNPE times include simulation, training, and 2,048-draw inference. For 100 different model specifications, these fixed-model baselines require 100 model-specific training procedures per method; AFIN reuses one network.
A Gaussian need not represent every posterior feature to be a useful proposal. AFIN learns forward KL across models, which penalizes missing posterior mass. NPE also learns forward KL, but requires the model-specific training above. The Gaussian decoder is a basic implementation of our factor architecture that makes both sampling and density evaluation inexpensive.
For integrable posterior expectations, standard SNIS is consistent under independent sampling when the proposal density is positive wherever the posterior density is positive [4]. AFIN's nonsingular Gaussian has full support. In Figure 4 [5], constructing the AFIN proposal and processing 500 million SNIS draws takes approximately three seconds per task. Results average ten sampling streams on 48 data-scarce and 36 heterogeneous tasks.
To compare proposal quality under the same SNIS budget, we include full-rank Gaussian variational inference (FRVI) and variational online Newton (VON) [6], a fast second-order Gaussian VI method. Both optimize reverse KL separately for each task.
AFIN achieves lower average corrected SW2, mean, and covariance errors than every displayed VON/FRVI setting at this budget. Times include proposal construction and correction. SNIS itself is standard; the same trained AFIN supplies these proposals across supported model changes without refitting. These results concern the evaluated dimensions; SNIS can still become inefficient as dimension grows.
Our problem setting assumes a known model specified by its factor types and numerical parameters. This is explicit in the submitted Problem Setup. Training uses joint simulations without posterior references, and direct prediction uses the trained network. Optional SNIS additionally evaluates the unnormalized posterior; this correction therefore requires evaluable densities, unlike purely simulator-based inference.
Long, well-mixed NUTS runs provide accurate inference; our advantage is the cost of reaching useful accuracy. All methods target the same posterior. NUTS has finite-sample error, whereas Gaussian VI also has approximation and optimization error; converging in reverse KL does not imply exact posterior moments or minimal SW2.
Revised Figure 2 [7] and Figure 3 [8] include NumPyro. At the displayed endpoints on the data-rich and intermediate synthetic panels, AFIN+SNIS achieves lower average SW2 with 8.4× and 10.1× faster inference, respectively. Longer NUTS runs are more accurate on the data-scarce and heterogeneous panels. Our claim is a useful direct estimate and inexpensive refinement at finite budgets, not universal superiority to long-running NUTS.
Each of the 144 synthetic reference posteriors uses four independent NumPyro NUTS chains, each with 10,000 warmup iterations followed by 1,000,000 retained samples (4 million retained samples per task). The worst R-hat is 1.0000103, minimum bulk ESS exceeds 2.43 million, and divergences are zero. On 30 conjugate tasks, NumPyro agrees with analytic posteriors to mean error 0.00063 and relative covariance error 0.00261.