acceptodds
Under review as a conference paper at ICLR 2027

Amortized Factor Inference Networks for Posterior Inference

Abstract

Amortized inference promises fast test-time Bayesian inference, but existing methods typically target a fixed model or a restricted model family. Extending amortization to new model families often requires retraining or test-time fine-tuning. In this paper, we ask: is it possible to build a single inference network capable of generalizing across varying priors, likelihoods, latent dimensions, and observation counts? We introduce Amortized Factor Inference Networks (AFINs), which describe a model as a list of typed prior and likelihood factors and map this description, together with the observations, to a Gaussian posterior approximation through an encode–merge–decode network whose learned parameters do not depend on the latent dimension or the number of observations. Experiments on synthetic and real-data benchmarks show that a single trained AFIN provides competitive posterior approximations at substantially lower test-time cost than NUTS and per-task variational inference. Optional importance sampling can further improve accuracy.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Amortized Bayesian inference, Posterior Inference, Importance sampling

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_Mr4SOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes amortized factor inference networks (AFIN), which encode priors and likelihoods as factors with known types and parameters. Through shared encoding, factor attention mechanisms, and a decoding network, it generates a Gaussian posterior approximation, which can be further refined using the SNIS method. Synthetic and real-data experiments show its accuracy and low inference costs.

Critical strengths
  1. The authors utilized the structure of the posterior to propose a pre-trained network that covers multiple prior-likelihood combinations, which can be used to quickly producing usable approximations.

  2. The architecture handles the product structure of the factors, allowing the same weights to accommodate different parameter settings, latent dimensions, and observation counts.

Critical weaknesses
  1. Neural posterior estimation already offers flexible approximations through Gaussian mixtures [1] and normalizing flows [2,3]. AFIN instead uses a single Gaussian with a strictly diagonally dominant precision matrix. This restricts its flexibility. Although the authors suggest that self-normalized importance sampling (SNIS) can be used for correction, which is a common adjustment method in Monte Carlo sampling (Also, SNIS may affected by the curse of dimensionality). With SNIS, I feel more like AFIN provides a proposal distribution for posterior sampling.

  2. The SNIS correction requires likelihood evaluation. However, many popular simulation-based inference methods, including neural posterior estimation, neural likelihood estimation, and neural ratio estimation, require only the ability to simulate from the likelihood [4].


[1]. Papamakarios and Murray. "Fast ε-free inference of simulation models with bayesian conditional density estimation", 2016.

[2]. Durkan, et al. "Neural spline flows", 2019.

[3]. Greenberg et al., "Automatic posterior transformation for likelihood-free inference", 2019

[4]. Lueckmann, et al. "Benchmarking simulation-based inference", 2021.

Questions for authors
  1. In Fig. 2 and Fig. 3, AFIN achieves lower Mean L2 and Sliced W2, but other methods like NUTS and VI do not seem to yield good results even after convergence. Is this because the runtime was insufficient, or do NUTS and VI fail on this type of problem, or are they targeting the wrong distribution?

  2. The training method (see lines 314-321) only on combinations of three priors and four likelihoods. If there are only a limited number of such combinations, how much additional benefit does the specialized factor architecture provide compared to a standard conditional neural posterior estimation that takes the same input information? If, in practical applications, the model has a complex likelihood (for example, Generalized linear model, but the likelihood is not among the types used during training), will the model still demonstrate excellent performance?

2 comments

Thank you for your review. We revised selected manuscript figures and added NPE and matched SNIS comparisons to clarify the accuracy, cost, and role of our Gaussian decoder.

W1. Gaussian flexibility and SNIS.

Flow-based NPE is more expressive, but fitted accuracy and per-model training cost also matter. We compared NPE [1] and three-round SNPE-C [2] with neural spline flows using sbi v0.26.1 [3], 100,000 simulations per model setup, and one training seed. Each panel contains six tasks, with two per synthetic difficulty. These are direct predictions without SNIS; SW2\mathrm{SW}_2 denotes sliced Wasserstein-2 distance.

MethodSynthetic SW2\mathrm{SW}_2 ↓Time (s)UCI SW2\mathrm{SW}_2 ↓Time (s)
AFIN0.04630.01100.05680.0122
NPE-NSF0.1233300.20.2365296.4
SNPE-C-NSF0.1418700.70.2814672.6

AFIN has lower average direct error on both panels, in about 0.01 seconds versus 296–701 seconds per task. NPE/SNPE times include simulation, training, and 2,048-draw inference. For 100 different model specifications, these fixed-model baselines require 100 model-specific training procedures per method; AFIN reuses one network.

A Gaussian need not represent every posterior feature to be a useful proposal. AFIN learns forward KL across models, which penalizes missing posterior mass. NPE also learns forward KL, but requires the model-specific training above. The Gaussian decoder is a basic implementation of our factor architecture that makes both sampling and density evaluation inexpensive.

For integrable posterior expectations, standard SNIS is consistent under independent sampling when the proposal density is positive wherever the posterior density is positive [4]. AFIN's nonsingular Gaussian has full support. In Figure 4 [5], constructing the AFIN proposal and processing 500 million SNIS draws takes approximately three seconds per task. Results average ten sampling streams on 48 data-scarce and 36 heterogeneous tasks.

To compare proposal quality under the same SNIS budget, we include full-rank Gaussian variational inference (FRVI) and variational online Newton (VON) [6], a fast second-order Gaussian VI method. Both optimize reverse KL separately for each task.

MethodData-scarce SW2\mathrm{SW}_2 ↓Time (s/task)Heterogeneous SW2\mathrm{SW}_2 ↓Time (s/task)
AFIN+SNIS0.01302.9890.00813.108
VON-10+SNIS0.01563.0120.01073.130
VON-30+SNIS0.01733.0730.01083.199
VON-100+SNIS0.01643.2930.01023.434
FRVI-30k+SNIS0.017522.9630.010525.717

AFIN achieves lower average corrected SW2\mathrm{SW}_2, mean, and covariance errors than every displayed VON/FRVI setting at this budget. Times include proposal construction and correction. SNIS itself is standard; the same trained AFIN supplies these proposals across supported model changes without refitting. These results concern the evaluated dimensions; SNIS can still become inefficient as dimension grows.

W2. Likelihood evaluation.

Our problem setting assumes a known model specified by its factor types and numerical parameters. This is explicit in the submitted Problem Setup. Training uses joint simulations without posterior references, and direct prediction uses the trained network. Optional SNIS additionally evaluates the unnormalized posterior; this correction therefore requires evaluable densities, unlike purely simulator-based inference.

Q1. Why do NUTS and VI retain error?

Long, well-mixed NUTS runs provide accurate inference; our advantage is the cost of reaching useful accuracy. All methods target the same posterior. NUTS has finite-sample error, whereas Gaussian VI also has approximation and optimization error; converging in reverse KL does not imply exact posterior moments or minimal SW2\mathrm{SW}_2.

Revised Figure 2 [7] and Figure 3 [8] include NumPyro. At the displayed endpoints on the data-rich and intermediate synthetic panels, AFIN+SNIS achieves lower average SW2\mathrm{SW}_2 with 8.4× and 10.1× faster inference, respectively. Longer NUTS runs are more accurate on the data-scarce and heterogeneous panels. Our claim is a useful direct estimate and inexpensive refinement at finite budgets, not universal superiority to long-running NUTS.

Each of the 144 synthetic reference posteriors uses four independent NumPyro NUTS chains, each with 10,000 warmup iterations followed by 1,000,000 retained samples (4 million retained samples per task). The worst R-hat is 1.0000103, minimum bulk ESS exceeds 2.43 million, and divergences are zero. On 30 conjugate tasks, NumPyro agrees with analytic posteriors to mean error 0.00063 and relative covariance error 0.00261.

Q2. Model diversity, conditional NPE, and unseen likelihoods.

A small factor library still defines a nontrivial inference family. With PP prior configurations, LL likelihood forms, and NN observations, there are P[binom(N+L−1,L−1)−L]P[\mathrm{binom}(N+L-1,L-1)-L] unordered heterogeneous type-count configurations, where binom is the binomial coefficient. For example, allowing up to five likelihood forms per task gives 744,174,320 possible configurations at P=4P=4, L=5L=5, D=16D=16, N=256N=256, before varying numerical parameters. These are possible supported inputs. Our model class includes standard Bayesian GLMs and also permits different supported likelihood families within the same task. In our heterogeneous benchmark with D=16D=16 and N=8N=8, eight scalar linear predictors leave at least eight latent directions unconstrained by the likelihood. With Student-t priors, the posterior retains polynomial tails along these directions.

The paper's ablation compares attention-based merging with sum pooling. Keeping node-and-pair representations, factor adapters, and the Gaussian decoder, Table 19 reports direct SW2\mathrm{SW}_2 0.1324 for sum pooling versus 0.0412 for AFIN under the same training simulator and schedule. The sum-pooling variant is smaller, so this comparison reflects both architecture and capacity.

New likelihood types can be added by training an adapter. The submitted Poisson extension evaluates 24 tasks and three seeds:

New-type trainingDirect SW2\mathrm{SW}_2 ↓SNIS SW2\mathrm{SW}_2 ↓Time (GPU-seconds)
Adapter only0.05910.030117,820
Joint fine-tuning0.04580.030530,204

Training only the new adapter, with the shared network frozen, gives comparable SNIS accuracy with about 40% less GPU time than joint fine-tuning. This demonstrates an economical extension to a new likelihood; performance on other new families would need validation after adaptation.

References

[1] Papamakarios and Murray. Fast ε-free Inference of Simulation Models with Bayesian Conditional Density Estimation. NeurIPS, 2016. https://arxiv.org/abs/1605.06376

[2] Greenberg et al. Automatic Posterior Transformation for Likelihood-Free Inference. ICML, 2019. https://proceedings.mlr.press/v97/greenberg19a.html

[3] sbi v0.26.1; NPE_C implementation. https://github.com/sbi-dev/sbi

[4] Owen. Monte Carlo theory, methods and examples. Chapter 9, Theorem 9.2. https://artowen.su.domains/mc/Ch-var-is.pdf

[5] Revised Figure 4: SNIS comparisons on data-scarce and heterogeneous tasks. https://anonymous.4open.science/r/ICLR_AFIN-F43C/figures/figure4.pdf

[6] Khan and Rue. The Bayesian Learning Rule. JMLR, 2023. https://jmlr.org/papers/v24/22-0291.html

[7] Revised Figure 2: synthetic benchmarks. https://anonymous.4open.science/r/ICLR_AFIN-F43C/figures/figure2.pdf

[8] Revised Figure 3: UCI benchmarks. https://anonymous.4open.science/r/ICLR_AFIN-F43C/figures/figure3.pdf

Reviewer_NehVOpenReview ↗
Rating
4
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

AFIN learns a posterior approximation from an explicit list of typed prior and likelihood factors, and their observations. A shared permutation-equivariant network encode them, attention exchanges information across factors, and a decoder maps it to a Gaussian approximation's parameters.

Critical strengths
  • The paper's primary contributions lie in designing a scalable and practical architecture whose learned parameters do not depend on the latent dimension or the number of observations.
  • The submission includes a large suite of experiment results showing strong empirical gains of the framework
Critical weaknesses

NA

Questions for authors
  • How much error does the diagonal dominance restriction incur beyond the Gaussian approximation limitation? Can the authors provide a comparison on correlated designs, ideally against an unrestricted Gaussian decoder or a per-task Gaussian optimum? That would help separate the amortization error from the approximation error.
1 comment

Thank you for your positive assessment and useful suggestion. We updated selected manuscript figures and added the following analysis.

Q1. How much error does diagonal dominance add?

We compare per-task forward-KL-optimal Gaussians with and without diagonal dominance of the precision matrix. Across 144 benchmark tasks, the constraint adds 0.000273 nats per latent dimension on average; AFIN's prediction adds another 0.003717 above the constrained optimum. On 108/144 tasks, the constraint adds no loss.

On correlated designs, two Gaussian-prior tasks with eight latent dimensions and 96 observations show additional constraint loss of 0.0162 nats per dimension at design-covariance condition number 10, and 0.0579 at 40.

Together with AFIN's empirical performance, these results suggest that diagonal dominance is not a major source of error on our benchmark.

Reviewer_yWVdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2(+1)
Read review
Paper summary

The paper proposes Amortized Factor Inference Networks (AFIN), a neural posterior estimator that is amortized over Bayesian models with a global latent, a prior factor, and NN conditionally independent likelihood factors, each of a type from a predefined set. One trained network handles varying DD, NN, prior and likelihood families, factor parameters, and observations. Its architecture is motivated by conjugate updates and each factor is encoded into node and pair features, the factors interact through attention, and their sum is decoded into a Gaussian variational posterior. Weights are shared across latent coordinates, so the parameter count is independent of DD and NN. Training minimizes expected forward KL on simulated tasks, and the variational posterior output is used directly or as a SNIS proposal. Experiments on synthetic tasks, UCI datasets, extrapolation in DD and NN, and a Poisson-likelihood extension compare against NUTS, VI, and NPE-PFN.

Critical strengths
  • The proposed method, AFIN, is interesting and tackles a novel problem in simulation-based inference in that AFIN can handle varying priors, likelihoods, latent dimensions and observation counts in a single amortization model.
  • The architecture with its encode, merge, decode structure is a sensible approach to the problem and takes inspiration from natural parameters in conjugate posterior updating.
  • The authors provide their code implementation.
Critical weaknesses
  • Every benchmark in the paper is a Bayesian GLM with one global latent vector and conditionally independent observations, so the posteriors are unimodal and almost Gaussian. There are no multimodal posteriors, no heavy-tailed or funnel-shaped nor symmetric posteriors. The experiments thus cannot show whether AFIN can handle nontrivial posterior geometry or mainly provides faster inference for simple GLM posteriors.
  • Speaking of faster inference: The per-task baselines are very weak for the low dimensional setting considered in the paper with D≤16D \leq 16 as FRVI uses Adam with reparameterization gradients but second-order methods would probably converge in a single or a handful of iterations, for instance variational online Newton [1], a Laplace approximation via Newton's method, or Pathfinder [2]. The wall-clock advantage is therefore likely overstated. Yet, cost is a central claim on the abstract and the contributions.
  • The training range (D≤16D \le 16, N≤256N \le 256) is small relative to the settting where NUTS is costly. Extrapolation in DD already worsens accuracy and SNIS increases covariance error there, so the advantage over NUTS or unadjusted samplers like MCLMC [3] is likely less than claimed.
  • There seems to be a heavy reliance on the importance sampling on top of AFIN and the submission does not differentiate between claims about the method itself and claims about the combination of the method with SNIS. The UCI experiment conclusion states that that AFIN + SNIS has "lower error than every baseline [...] on all 12 tasks", but this compares against baselines without importance sampling on top. Baseline + SNIS curves are needed for a meaningful comparison (especially since the AFIN (single shot) estimate does not outperform any baseline on its own).
  • Interestingly, AFIN is framed as a simulation-based inference method but baselines from SBI are completely missing from the experimental section of the paper (there is a comparison against NPE-PFN but this is a niche method. Also, only marginal coverage of the AFIN posteriors averaged over coordinates and tasks is reported, SBC rank statistics, per-dimension coverage or a predictive evaluation on UCI (e.g., using the LPPD) are all missing.
  • Minor remark: Table 1 states that factor types tt are changeable "without retraining" but this only apploes to seen types, unseen types still require adapter retraining as far as I understand.

References

[1] Khan, M. E., & Rue, H. (2023). The Bayesian learning rule. Journal of Machine Learning Research, 24(281), 1-46.

[2] Zhang, L., Carpenter, B., Gelman, A., & Vehtari, A. (2022). Pathfinder: Parallel quasi-Newton variational inference. Journal of Machine Learning Research, 23(306), 1-49.

[3] Robnik, J., & Seljak, U. (2023). Fluctuation without dissipation: Microcanonical langevin monte carlo. In Sixth Symposium on Advances in Approximate Bayesian Inference-Archival Track.

Questions for authors
  1. Does the NumPyro NUTS timing include JIT compilation? How is the timing experiment and measurement fair given that AFIN's own compilation cost is excluded from time measurements?
  2. Can you elaborate how AFIN compares to appropriate baselines, such as variational online Newton [1], a Laplace approximation via Newton's method, and Pathfinder [2]? Also, can you show how AFIN + SNIS compares to these baselines with added post-hoc importance sampling (multi-path Pathfinder already somewhat does this internally)?
  3. Are there advantages to using AFIN (without and with SNIS) over per-task (variational) inference?
  4. Non-Gaussian posterior geometry can be produced with the factor types you already support. For example, logistic regression on separable data. How do AFIN and AFIN + SNIS perform there, and what is k^\hat k?
  5. Why do you time NUTS in Pyro when NumPyro is reportedly about 25x faster?
  6. What does the Figure 2 comparison look like when the D=16,N=1D=16, N=1 setting is excluded as it is dominated by the prior?
  7. How does AFIN compare to SBI NPE methods?
14 comments

Thank you for the detailed review and helpful baseline suggestions. We revised Figures 2 [7], 3 [8], and 4 [9] and added the comparisons and diagnostics below.

Following your suggestion, we have added variational online Newton (VON) [1] to the revised manuscript, including matched VON+SNIS comparisons in Figure 4 [9].

W1. Are all benchmarks almost-Gaussian GLMs?

A shared latent and conditional independence do not imply an almost-Gaussian posterior. The submitted Forest Fires and Mesquite tasks mix continuous and Bernoulli likelihoods within one model, extending beyond a standard single-family GLM. In our synthetic heterogeneous benchmark with D=16D=16 and N=8N=8, eight scalar linear predictors leave at least eight latent directions unconstrained by the likelihood. With Student-t priors, the posterior retains polynomial tails along these directions. These heterogeneous cases combine different likelihood families while retaining tail behavior that no Gaussian approximation can reproduce. Figure 4 [9] includes these cases.

W2. Faster second-order baselines.

Thank you for suggesting VON; it is indeed a strong, fast baseline. Ten updates take approximately 0.034 seconds, compared with AFIN's 0.011 and FRVI-30k's 20–23.

Better direct accuracy does not necessarily mean a better SNIS proposal. On heterogeneous tasks, VON-10 achieves lower average direct SW2\mathrm{SW}_2 than AFIN (0.0429 versus 0.0542), although it requires more time. Yet after 500 million SNIS draws, the ordering reverses: AFIN reaches 0.0081 versus VON-10's 0.0107 (Figure 4 [9]). VON [1] optimizes the same reverse-KL objective as FRVI more efficiently, but this objective can underrepresent posterior regions that matter for importance sampling. AFIN's forward-KL training encourages mass coverage. The observed finite-budget gap is consistent with this distinction; Q2 provides the full comparison.

W3. Training range and extrapolation.

Amortizing this bounded model family is nontrivial. With PP prior configurations, LL likelihood forms, and NN observations, there are P[binom(N+L−1,L−1)−L]P[\mathrm{binom}(N+L-1,L-1)-L] unordered heterogeneous type-count configurations; binom is the binomial coefficient, and subtracting LL excludes homogeneous cases. For example, allowing up to five likelihood forms per task gives 744,174,320 possible configurations at P=4P=4, L=5L=5, D=16D=16, N=256N=256, before varying numerical parameters. One network accepts these supported combinations and varying DD and NN. We agree that extrapolation can reduce accuracy, including after SNIS; our claim concerns reuse within the supported range.

W4. Direct inference and equally corrected baselines.

We added the requested NSF-VI+SNIS and MAF-VI+SNIS comparisons. All methods receive one million proposals on the 12 UCI tasks in Figure 3 [8]. SW2\mathrm{SW}_2 denotes sliced Wasserstein-2 distance.

MethodDirect SW2\mathrm{SW}_2 ↓+SNIS SW2\mathrm{SW}_2 ↓Time (s)
AFIN0.071570.001450.032
NSF-VI0.037360.0180362.233
MAF-VI0.028710.0013382.678

AFIN+SNIS has lower average error than NSF+SNIS and nearly matches MAF+SNIS, taking approximately 1/2,600 of MAF's time. Times include fitting and correction, averaging all three flow fit seeds.

W5. SBI baselines and posterior diagnostics.

We added rank diagnostics and predictive evaluations, and summarize average coverage below. NPE comparisons are in Q7.

SBC rank statistics. Across 120 prior-predictive tasks within the training range, the maximum rank-CDF deviation from uniformity is 0.046 for AFIN and 0.037 after SNIS. We use randomized ranks for direct AFIN and weighted posterior CDFs for SNIS.

Coverage. For nominal 90% credible intervals, average coverage is 87.5% for AFIN and 87.1% after SNIS. Their 95% task-bootstrap intervals are 84.1–90.5% and 83.7–90.2%, respectively; both include the nominal 90%.

Predictive evaluation. We evaluated held-out predictions on five classification and five regression datasets, with three splits each:

Held-out metricAFINAFIN+SNIS
Classification NLL ↓0.47980.4836
Regression log predictive density ↑-0.7634-0.7634

NLL denotes negative log predictive density. Both predictive scores change little after SNIS, indicating that direct AFIN already gives similar predictions to its corrected version.

W6. Unseen factor types require retraining.

Yes; “without retraining” applies to supported factor types. We have clarified this in Table 1. For an unseen type, the submitted Poisson experiment trains a new adapter while keeping the shared network frozen.

Q1. NumPyro compilation and timing fairness.

Yes, NumPyro timings include JIT compilation, warmup, and sampling. A practitioner may specify a Bayesian model, run inference, then try another prior or likelihood family, change the latent dimension, or use more observations. Changes to model structure or input shapes generally require recompilation, followed by a new NUTS run. Our timing includes the compilation and inference costs incurred in this model-development workflow. AFIN is loaded once and immediately reused across these supported model changes, without per-task fitting.

Q2. VON, Laplace and Pathfinder, including SNIS.

AFIN has the lowest average corrected error in both panels. All rows use 500 million proposals and ten SNIS streams, over 48 data-scarce and 36 heterogeneous tasks. Laplace uses matched subsets of 36 and 27 tasks with smooth priors.

Proposal (+SNIS)Data-scarce SW2\mathrm{SW}_2 ↓Time (s)Heterogeneous SW2\mathrm{SW}_2 ↓Time (s)
AFIN0.01302.9890.00813.108
VON-100.01563.0120.01073.130
VON-300.01733.0730.01083.199
VON-1000.01643.2930.01023.434
FRVI-30k0.017522.9630.010525.717
Pathfinder0.14642.9970.12443.121
Pathfinder mixture (4 paths)0.13146.8510.09646.988
AFIN, Laplace-eligible0.01302.9930.00793.113
Laplace, eligible tasks0.02492.9830.01943.104

AFIN reduces error by 17% and 21% relative to the best VON setting in each panel, at similar runtime. More VON iterations do not close the gap in Figure 4 [9]. Times include fitting and correction; Laplace/Pathfinder use warm fits. The four-path Pathfinder [6] row uses an equal Gaussian mixture with ordinary SNIS. Its largest errors occur with nonsmooth priors; AFIN also improves on smooth Student-t cases.

Q3. Advantages over per-task VI.

AFIN learns forward KL from simulations and predicts without per-task optimization. For posterior π\pi and approximation qq, VON/FRVI optimize reverse KL, KL(q∥π)\mathrm{KL}(q\Vert\pi). AFIN learns KL(π∥q)\mathrm{KL}(\pi\Vert q), averaged across models, which penalizes missing posterior mass even with a Gaussian output. Figure 4 [9] shows the resulting proposal's value for cheap SNIS; Q7 evaluates direct inference. For 100 different model specifications, per-task VI requires 100 fits; AFIN needs 100 forward passes through the same trained network.

Q4. Separable logistic regression and k-hat.

On separable data, SNIS reduces AFIN's average SW2\mathrm{SW}_2 by 76% at N=16N=16 and 89% at N=96N=96, in about 0.02 seconds. We tested six datasets at D=8D=8. Rows average Gaussian, Laplace, and Student-t priors, using one million proposals and three SNIS streams. Time includes construction and correction.

NNMethodDirect SW2\mathrm{SW}_2 ↓+SNIS SW2\mathrm{SW}_2 ↓Mean k-hatTime (s)
16AFIN0.05300.01290.5170.020
16VON-100.08050.02410.5160.031
16VON-300.06830.02310.5200.076
16VON-1000.06600.02100.5070.230
96AFIN0.26940.03090.4400.019
96VON-100.35580.07490.5610.034
96VON-300.06820.01250.4450.082
96VON-1000.06420.01250.4560.243

At N=16N=16, AFIN+SNIS is more accurate and faster than every VON setting tested. At N=96N=96, VON-30/100 is more accurate but takes 4–13× as long. The tail diagnostic k-hat [5] is below 0.7 for five of the six AFIN task averages; the Student-t/N=96N=96 case is borderline at 0.707.

Q5. Pyro versus NumPyro.

We have added NumPyro NUTS to both the synthetic and UCI comparisons. Pyro was originally chosen for integration with our PyTorch implementation. Revised Figures 2 [7] and 3 [8] now compare both backends on the same targets, with four chains and matched sampler settings. Q6 reports the advantage directly against NumPyro.

Q6. Excluding D=16D=16,N=1N=1.

The advantage remains across the other 96 homogeneous tasks. Figure 2 [7] gives these averages over 48 tasks per regime:

RegimeAFIN+SNIS SW2\mathrm{SW}_2 ↓Time (s/task)NumPyro SW2\mathrm{SW}_2 ↓Time (s/task)
Data-rich (D=4D=4,N=256N=256)0.000614.9440.0008141.456
Intermediate (D=8D=8,N=64N=64)0.001522.5840.0019126.129

AFIN+SNIS has lower SW2\mathrm{SW}_2 and 8.4×/10.1× faster inference, so the gains are not confined to D=16D=16,N=1N=1. These endpoints use 500 million SNIS proposals and 100,000 retained NumPyro draws per chain.

Q7. Comparison with standard NPE.

AFIN achieves lower average direct error on both panels. NPE [2] and three-round SNPE-C [3] use sbi [4], spline flows, 100,000 simulations per model setup, and one training seed. Each panel has six tasks, with two per synthetic difficulty. No SNIS is used.

MethodSynthetic SW2\mathrm{SW}_2 ↓Time (s)UCI SW2\mathrm{SW}_2 ↓Time (s)
AFIN0.04630.01100.05680.0122
NPE-NSF0.1233300.20.2365296.4
SNPE-C-NSF0.1418700.70.2814672.6

AFIN reduces average SW2\mathrm{SW}_2 by 62–67% on synthetic tasks and 76–80% on UCI, taking approximately 0.01 seconds versus 296–701 seconds per task. NPE/SNPE time includes simulation, training, and 2,048-draw inference. NPE also learns forward KL, but trains separately for each model setup here; AFIN reuses one network across supported model revisions.

References

[1] Khan and Rue. The Bayesian Learning Rule. JMLR, 2023. https://jmlr.org/papers/v24/22-0291.html

[2] Papamakarios and Murray. Fast ε-free Inference of Simulation Models with Bayesian Conditional Density Estimation. NeurIPS, 2016. https://arxiv.org/abs/1605.06376

[3] Greenberg et al. Automatic Posterior Transformation for Likelihood-Free Inference. ICML, 2019. https://proceedings.mlr.press/v97/greenberg19a.html

[4] sbi v0.26.1; NPE_C implementation. https://github.com/sbi-dev/sbi

[5] Vehtari et al. Pareto Smoothed Importance Sampling. JMLR, 2024. https://www.jmlr.org/papers/v25/19-556.html

[6] Zhang et al. Pathfinder: Parallel quasi-Newton variational inference. JMLR, 2022. https://jmlr.org/papers/v23/21-0889.html

[7] Revised Figure 2: synthetic benchmarks. https://anonymous.4open.science/r/ICLR_AFIN-F43C/figures/figure2.pdf

[8] Revised Figure 3: UCI benchmarks. https://anonymous.4open.science/r/ICLR_AFIN-F43C/figures/figure3.pdf

[9] Revised Figure 4: SNIS comparisons on data-scarce and heterogeneous tasks. https://anonymous.4open.science/r/ICLR_AFIN-F43C/figures/figure4.pdf

Reviewer_aArqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Amortized Factor Inference Networks (AFINs). An AFIN is a neural posterior estimator whose inputs are the observations and a description of the model. The model is described as a list of typed factors: one prior factor and one likelihood factor per observation. Each factor is given by a type label from a fixed vocabulary and its known numerical parameters. The architecture has three parts, modeled on conjugate updating: type-specific adapters and a shared network encode each factor into node (D×C) and pair (D×D×C) embeddings; Transformer blocks over factors, without positional encodings, let the embeddings interact before they are summed; and a decoder outputs a Gaussian with a strictly diagonally dominant precision matrix.

All maps are parameter-tied across latent coordinates, so the learned parameters do not depend on the latent dimension D or the number of observations N.

Critical strengths

The idea of amortizing both over the data input (that has been done extensive in the literature) as well as models is new and interesting.

The framework is very general which can handle inputs of different length, dimension and different type of inputs. Basically, it is very easy to add new factor types (prior or likelihood)

Critical weaknesses

It seems the model class is restricted to a single global latent, conditionally independent observations and fully known factor parameters, which seems quite restrictive .

Eq. 2 is standard NPE on a larger simulator in which m is part of the conditioning input; it would help to explain how this differs from ordinary NPE. The complexity would still blow up?

The paper lacks any kind of theoretical analysis which to me is a critical weakness. I was really hoping to see any theoretical demonstration of improvement of this current framework versus a traditional fixed model simulation based inference even for some simple setup. Conceptually speaking, if the true data is simulated from a model that's far away from a fixed simulator, the current framework that allows amortization over different models should suffer less from simulator misspecification?

Now looking at the model used for numerical studies, it seems the benchmarked likelihood depends on z only through xᵀz, and the adapter descriptors (e.g., y·x_i and x_i·x_j) are tailored to this structure. This seems quite simple and narrow.

The UCI tasks use the same model family. Covariates are standardized and whitened and targets are standardized, which aligns the tasks with the training distribution. So the UCI experiments do not test model types beyond those seen in training

Questions for authors

See above

4 comments

Thank you for your review. We revised selected manuscript figures and added comparisons to clarify the contribution and address your questions.

W1. Restricted model class.

The factorization is restricted, but inference across this family is nontrivial. Our model class includes standard Bayesian GLMs and also supports heterogeneous likelihoods: each of the NN observation factors can have its own supported likelihood type and numerical parameters within the same task. For example, in our heterogeneous benchmark with D=16D=16 and N=8N=8, the likelihood depends on the latent vector through eight scalar linear predictors, leaving at least eight directions unconstrained by the likelihood. With Student-t priors, the posterior retains polynomial tails along these directions.

For PP prior configurations and LL likelihood forms, there are P[binom(N+L−1,L−1)−L]P[\mathrm{binom}(N+L-1,L-1)-L] unordered heterogeneous type-count configurations; binom denotes the binomial coefficient. For example, even allowing at most five likelihood forms per task, P=4P=4, L=5L=5, D=16D=16,N=256N=256 permits 744,174,320 possible configurations, before varying numerical parameters. A practitioner can revise a supported prior, likelihood mixture, or dimension and immediately repeat inference with the same network.

W2. Relation to ordinary NPE and complexity.

Eq. 2 uses the standard conditional-NPE objective; our contribution is the architecture that makes inference reusable across model specifications. Type-specific adapters and shared factor interactions handle different model inputs, while parameter tying supports varying DD and NN without changing the learned parameter shapes. We sample model configurations during training and share one network across them. Broader model coverage can require more training.

For example, 100 distinct model specifications require 100 separate NPE training runs with our fixed-model baselines. Each run incurs the optimization cost and the challenge of obtaining an accurate posterior fit with a flexible density estimator. AFIN reuses the same trained network across supported model changes, without per-model optimization.

Per-task computation still grows with DD and NN; Appendix B.5 gives the computational costs, and Appendix E.4 reports measured runtime and memory.

We tested both accuracy and the cost of model-specific training. We compared NPE [1] and three-round SNPE-C [2] using sbi v0.26.1 [3], neural spline flows, 100,000 simulations per model setup, and one training seed. M1\mathrm{M}_1/M2\mathrm{M}_2 are mean/covariance errors; SW2\mathrm{SW}_2 is sliced Wasserstein-2 distance.

Synthetic: six tasks, two per difficulty; no SNIS

MethodDirect M1\mathrm{M}_1 ↓Direct M2\mathrm{M}_2 ↓Direct SW2\mathrm{SW}_2 ↓Time (s/task) ↓
AFIN0.08500.24520.04630.0110
NPE-NSF0.08920.27610.1233300.2
SNPE-C-NSF0.10880.30290.1418700.7

UCI: six tasks; no SNIS

MethodDirect M1\mathrm{M}_1 ↓Direct M2\mathrm{M}_2 ↓Direct SW2\mathrm{SW}_2 ↓Time (s/task) ↓
AFIN0.13710.06580.05680.0122
NPE-NSF0.30870.33640.2365296.4
SNPE-C-NSF0.37720.49290.2814672.6

AFIN achieves lower average mean, covariance, and SW2\mathrm{SW}_2 errors on both panels, in approximately 0.01 seconds per task. Time includes simulation, training, and 2,048-draw inference for NPE/SNPE.

W3. Theory relative to fixed-model NPE.

We do not claim a theoretical contribution; standard results provide the requested model-mismatch example and an SNIS error bound. Let z∼N(0,1)z\sim\mathcal{N}(0,1) and y∣z,s∼N(sz,1)y\mid z,s\sim\mathcal{N}(sz,1), where the known model parameter s∈{−1,+1}s\in\lbrace-1,+1\rbrace. Gaussian conditioning gives posterior N(sy/2,1/2)\mathcal{N}(sy/2,1/2), with variance as the second argument. Exact NPE trained for s=+1s=+1 incurs forward-KL error y2y^2 when reused for s=−1s=-1. An estimator conditioned on ss can represent both posteriors exactly.

Existing SNIS theory connects proposal quality to refinement cost. Let π\pi be the posterior, qq a proposal positive wherever π\pi is positive, and KK the number of independent draws from qq. For any function hh with ∣h(z)∣≤1\lvert h(z)\rvert\le1, the SNIS mean-squared error for Eπ[h]\mathbb{E}_{\pi}[h] is bounded by 4[1+χ2(π∥q)]/K4[1+\chi^2(\pi\Vert q)]/K, provided χ2(π∥q)=∫π(z)2q(z)dz−1\chi^2(\pi\Vert q)=\int\frac{\pi(z)^2}{q(z)}dz-1 is finite [4]. AFIN makes large KK inexpensive through cheap sampling and density evaluation. Forward-KL training alone does not ensure finite χ2\chi^2; Figure 4 [5] measures the benefit at matched finite budgets.

W4. Linear predictors and tailored descriptors.

Linear predictors do not imply Gaussian or conjugate posteriors. Gaussian, Student-t, and Bernoulli likelihood factors can coexist in one task and combine with a non-Gaussian prior. The descriptors encode the known local factors; the shared network learns how their contributions combine into the joint posterior. In the paper's node-and-pair ablation (Table 19), direct SW2\mathrm{SW}_2 is 0.1324 with sum pooling versus 0.0412 with AFIN's attention-based merging. Both use the same adapters, Gaussian decoder, training simulator, and schedule.

W5. UCI preprocessing and unseen model types.

UCI tests real-data transfer within supported families; new factor types use adapter training. All methods use the same tasks and preprocessing in Figure 3 [6]. We do not claim zero-shot inference for unsupported factor types. The submitted Poisson extension trains only the new adapter while freezing the shared network, achieving comparable SNIS accuracy to joint fine-tuning (SW2\mathrm{SW}_2 0.0301 versus 0.0305) with about 40% less GPU time.

References

[1] Papamakarios and Murray. Fast ε-free Inference of Simulation Models with Bayesian Conditional Density Estimation. NeurIPS, 2016. https://arxiv.org/abs/1605.06376

[2] Greenberg et al. Automatic Posterior Transformation for Likelihood-Free Inference. ICML, 2019. https://proceedings.mlr.press/v97/greenberg19a.html

[3] sbi v0.26.1; NPE_C implementation. https://github.com/sbi-dev/sbi

[4] Agapiou et al. Importance Sampling: Intrinsic Dimension and Computational Cost. Statistical Science, 2017, Theorem 2.1. https://arxiv.org/abs/1511.06196

[5] Revised Figure 4: SNIS comparisons on data-scarce and heterogeneous tasks. https://anonymous.4open.science/r/ICLR_AFIN-F43C/figures/figure4.pdf

[6] Revised Figure 3: UCI benchmarks. https://anonymous.4open.science/r/ICLR_AFIN-F43C/figures/figure3.pdf

AuthorsOpenReview ↗

Kind follow-up on our rebuttal response

As the rebuttal period ends tomorrow, we would be grateful if you could review our responses to each of your concerns and questions and share any further feedback. Thank you again for your time and thoughtful review!

AuthorsOpenReview ↗

Final Gentle Reminder on Our Rebuttal

Dear Reviewer aArq,

With less than 24 hours remaining in the discussion period, we would like to kindly follow up on our rebuttal. We believe we have addressed all your concerns and questions, and would greatly appreciate it if you could take a moment to review our responses and reconsider your assessment.

Thank you very much for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.