acceptodds
Under review as a conference paper at ICLR 2027

Prior-Amortized In-Context Bayesian Inference for Generalized Linear Mixed-Effects Models

Abstract

Hierarchical data is ubiquitous in the empirical sciences and most commonly analyzed with generalized linear mixed-effects models (GLMMs), a flexible hierarchical model class with interpretable parameters. Bayesian inference for GLMMs yields calibrated uncertainty estimates but requires Markov Chain Monte Carlo (MCMC); the gold-standard No-U-Turn Sampler (NUTS) is slow and must restart from scratch for every new dataset, model and prior specification. We introduce metabeta, a pretrained neural network for prior-amortized in-context Bayesian inference over GLMMs. Unlike previous neural posterior estimators that fix the prior at training time, metabeta accepts prior families and hyperparameters as direct inputs at test time, enabling zero-shot generalization to new prior specifications. Two set transformers and two conditional neural spline flows mirror the posterior's two-level structure (global parameters shared across groups, local parameters per group) and are trained jointly on millions of realistic simulated datasets with continuous, binary and count outcomes. By default, the flow posterior is refined by Independence Metropolis–Hastings against the unnormalized posterior; this yields tuning-free asymptotically correct inference two to three orders of magnitude faster than NUTS. Alternatively, metabeta warm-starts NUTS, yielding nearly identical inference 2–12× faster while removing over 98% of its divergent transitions. On controlled benchmarks with ground-truth parameters, metabeta matches NUTS in parameter recovery, posterior calibration and out-of-sample prediction; on real datasets with unknown parameters, priors and model structure, its posteriors closely agree with those of NUTS; and it remains faithful where amortized models typically struggle: under misspecified likelihoods, misspecified priors, out-of-distribution predictors, collinear designs and data-poor regimes. On real data, a full prior sensitivity analysis including model comparison takes a few seconds with metabeta against 7–17 hours for NUTS, tracing its posteriors throughout. Our model is open-source and open-weights and thus immediately deployable.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
neural posterior estimation, mcmc, glmm

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.67 over 3 reviews, above 99% of the others.
Reviewer_u7oaOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
2(+1)
Soundness
2
Read review
Paper summary

This paper proposes a pretrained framework for prior-amortized Bayesian inference in GLMMs to achieve fast inference.

Critical strengths
  1. The paper provides a comprehensive solution to the proposed problem, with extensive experiments that support its effectiveness.
Critical weaknesses
  1. The practical need of the proposed pretrained model is not sufficiently justified.
Questions for authors
  1. My major concern is the practical need of the proposed method. The paper shows computational gains over NUTS, but it remains unclear to what extent the computational cost of posterior inference is a bottleneck in empirical studies. In my opinion, research design, data collection, and model specification may be considerably more time-consuming. Could the authors identify concrete examples from published empirical studies where the computational cost of NUTS (or other Bayesian inference method for GLMMs) has substantially constrained the analysis, thereby motivating the need for this project?
  2. The first paragraph, as well as Appendix F.1, reports that about half of the surveyed papers use GLMMs. However, it is unclear what proportion actually use Bayesian inference. Since the proposed method specifically targets Bayesian GLMM inference, this is important for assessing its practical relevance.
  3. The writing could be improved, particularly in the Abstract, by reducing the use of “;” to connect independent sentences.
3 comments
AuthorsOpenReview ↗

Practical need, Bayesian share and readability

Thank you for the encouraging review. We are glad you see MetaBeta (MB) as "a comprehensive solution" and appreciate our "extensive experiments that support its effectiveness". All changes are marked green in the revised PDF.

Q1

In my opinion, research design, data collection, and model specification may be considerably more time-consuming. Could the authors identify concrete examples from published empirical studies where the computational cost of NUTS [...] has substantially constrained the analysis [...]?

Published studies report cutting their analyses because NUTS was too slow:

  • [1] cut its prior sensitivity analysis to "one very large prior [...] to once again reduce the compute time to manageable levels", and dropped a random effect to "drastically reduce the resource requirements and compute time".
  • [2] capped a simulation study of random slopes: "Because running brms models takes very long, I could not run more replications, or more MCMC samples. Simulation 3 already took several months of computation time."
  • [3] capped a Bayesian power analysis at 100 simulated datasets per sample size: "our simulations took some 40 hours, and this was with a computer with 40 physical cores [...] the Bayesian analysis is 120 times slower."

The revision cites more cases in Sec. 1 and App. G.1.

Two of the stages you name are themselves many-fit workflows: research design includes power analysis, which needs to refit the model on thousands of simulated datasets for GLMMs [4]. Model specification includes model comparison and prior sensitivity analysis. MB cuts these from days or hours to minutes or seconds:

workflowMBNUTSMB vs NUTS
model comparison (1,783 datasets, App. D.1)36 minutesover 27 core-dayssame Bayes-factor category: 98-99%
prior sensitivity analysis (54-132 priors, App. D.2)a few seconds14-18 core-hourssame result: 100%
Bayesian power analysis (20,000 posteriors, App. D.3)9 minutesabout 46 core-days (extrapolated)same decision: 98.7%

The power analysis is a new experiment (App. D.3). Suppose half of mm schools get an educational program, 2020 students per school pass or fail, and we study the effectiveness of the program with the model passed ~ program + (1 | school). How many schools does the study need in order to detect an effect? Following the standard simulation-based procedure [4, 5], we fit 1,000 simulated trials per study design under a weakly informative and a skeptical prior. Then we count how often the 95% credible interval of the effect excludes zero. MB's power curves match those of NUTS and show that 80% power requires 40-60 schools under either prior. Finding this out after sampling too few (or unnecessarily many) schools translates to substantial delays to conducting such a study.

Q2

[...] it is unclear what proportion actually use Bayesian inference.

About one in seven: 14% of the surveyed GLMM papers report a Bayesian analysis, and 12% of citations to GLMM software packages went to the Bayesian ones in 2025 (App. G.1). We believe the share would be higher if Bayesian inference were faster, since runtime is a common reason to avoid it: "A frequent objection that we encounter to the type of Bayesian analyses we have presented here [...] is that the workflow is very time-consuming compared to a frequentist analysis" [3].

Q3

The writing could be improved, particularly in the Abstract, by reducing the use of ";" to connect independent sentences.

Thanks for bringing this up. We dropped semicolons from the abstract to split independent sentences. We also split 14 other sentences in the main text in a similar spirit. Happy to take further pointers on hard to read sentences or passages.


Thank you again for pushing us to make the practical case more explicit. Does the evidence above address your concern about the practical need?

References

Reviewer_u7oaOpenReview ↗

Response to authors

I thank the authors for the responses and additional experiments. The presented examples demonstrate the practical value of the proposed method. These efforts address my main concern. The response has strengthened my assessment of the paper, and I have raised my rating.

Thank you very much for going through our response and for raising your score! We believe, your request has strengthened the practical argument of our paper.

Reviewer_YHwaOpenReview ↗
Rating
4
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces a PFN model for GLMM modeling. Interestingly, it combines the PFN with MCMC, and allows for conditioning on different type of GLMM model assumptions.

Critical strengths

The paper resolves a major open problem in amortization: it combines a fast method (PFN) with the exactness and assymptotic correctness of MCMC. This alone is a great advance which should inspire the broader field of ammortization greatly - because the amortized methods have had issues with misspecification limiting their use. To that end, the PFN is combined with flow modeling, to yield a multivariate posterior (so far, usually PFNs only target univariate outputs due to problems with getting things to work with multivariate outputs).

The paper analyzes a very popular and important technique for science; GLMMs, and develops a strong PFN model for them. It further proposes to build a toolbox for posterior analysis that is common in this field.

The transformer is even trained to deal with various types of priors out of the box; meanwhile, most PFN methods do not condition on multiple priors out of the box.

Critical weaknesses

The appendix is rather long; it would be nice if the authors more clearly reference to it in the main body (for example; figure B1 has an additional row compared to the main body, would be good to mention that in main).

It might be nice to describe how the features are sampled in section 2.1.

Questions for authors

How were the groups z decided? Which features are selected for them? (alpha)

MB was undefinied (first used in Table 1); should be metabeta

In section 3.6 ; was the model conditioned on the wrong prior; or was the prior outside the training distribution entirely? Did you evaluate both types of misspeficiation ?

1 comment
AuthorsOpenReview ↗

Pointers, random-effect design and out-of-support priors

Thank you for the positive review. We appreciate your assessment that MetaBeta "resolves a major open problem in amortization", and that you see it as "a great advance". The revised PDF marks all changes in green:

  • Each Appendix is now referenced in the main text.
  • Fig. 2 now has a pointer to App. C.1 (where performance plots additionally show the raw flow).
  • Predictor sampling is briefly explained under "Predictor sources" in Sec. 2.4. The detailed explanation is in App. G.4 and G.5.
  • MB is defined as the abbreviation of MetaBeta in Tab. 1 (where a reader would first encounter MB, your Q2).

Q1

How were the groups z decided? Which features are selected for them? (alpha)

The random-effect design Z\mathbf Z is the first qq columns of the fixed-effect design X\mathbf X, where qq is the number of random effects. This is without loss of generality, since the released code orders the columns of X\mathbf X to match a supplied model formula (App. G.2).

Q3

In section 3.6 ; was the model conditioned on the wrong prior; or was the prior outside the training distribution entirely? Did you evaluate both types of misspeficiation?

Good point, Sec. 3.6 did not originally state this. We found that for 3×3\times wider prior scales, priors already leave the training range (in 96% of Bernoulli and 94% of Poisson datasets; App. E.2).

We ran an additional experiment to deliberately place priors outside the training range (App. E.3). Prior scales outside the range remained largely unproblematic. However, extreme prior locations and highly skeptical prior scales led to meaningful disagreements with NUTS (percentages pooled over families):

  • Prior scales at 1.1 or 2 times the trained maximum: MB disagrees with NUTS on 2% of datasets.
  • Prior means shifted by 5 prior SDs: MB disagrees with NUTS on 20% of datasets.
  • Prior scales at half the trained minimum: MB disagrees with NUTS on 49% of datasets.

The safeguards caught every dataset where MB disagreed with NUTS. The released model now warns the user when a specified prior is outside of the training range (App. B.4).


Thank you again for the encouraging review. Happy to clarify anything else.

Reviewer_yUwzOpenReview ↗
Rating
4(+2)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper details the construction of an amortized neural posterior approximator for the class of generalized linear mixed-effect models (GLMM), describe how to bundle it together with MCMC sampling (both IMH and NUTS) and present experiments comparing with the gold-standard (NUTS) that they seek to replace. They find the full pipeline to reach gold-standard accuracy, while being much faster due to amortization, and publish code and weights under the name 'metabeta'.

Critical strengths

Critical strengths

The main novelty lies in pushing the amortization scope for a well defined and widely used class of Bayesian models towards what some might call a foundation model for GLMMs which is a significant milestone on the ABI research program by scaling previously known techniques further than previously tried in a well chosen and well specified model class.

The description of task, method and experiments is clear and the experiments are compelling as far as I was able to assess (although see question (3) below).

The speed improvement due to amortization is evident and of high relevance for Bayesian Workflow in GLMMs. The key question is whether it also attains the level of trust of the "current" gold standard. The comparison targets (NUTS and ADVI) are well chosen to illustrate where the the Pareto frontier is pushed.

The misspecification stress tests cover substantial ground and it is perhaps the most striking argument for trustworthiness, that "no stress we applied leads to a meaningful disagreement between metabeta and NUTS". See question (4) below.

Critical weaknesses

Critical weakness

However, the boundary between related work and novel contributions is shaky, as it neglects two of the three publications in the literature, which together arguably make the core idea of metabeta when applied to GLMMs:

  1. Claimed contribution 1 (Prior-amortized inference over a model class) is an execution on the recipe laid out in [1] applied to GLMMs.
  2. Claimed contribution 2 (Exactness of MCMC) is an execution on the recipe laid out in [2] applied to GLMMs.

(*: The third piece is Habermann et al. which is acknowledged.)

This misrepresentation would be ground for rejection, but I expect to change my review favorably when the placement of the present work in the literature are reworked to fix the over-claiming, since I maintain that the actual novel contributions are significant on their own to be of interest to the ICLR program and worthy of acceptance.

[1] Elsemüller, L., Olischläger, H., Schmitt, M., Bürkner, P.-C., Koethe, U., & Radev, S. T. (2024). Sensitivity-Aware Amortized Bayesian Inference. Transactions on Machine Learning Research. https://openreview.net/forum?id=Kxtpa9rvM0

[2] Li, C., Vehtari, A., Bürkner, P.-C., Radev, S. T., Acerbi, L., & Schmitt, M. (2025). Amortized Bayesian Workflow. Transactions on Machine Learning Research. https://openreview.net/forum?id=osV7adJlKD

Questions for authors

Questions

  1. Why is IMH the default, not NUTS warm-starting, is it mainly speed? What is the workflow you propose people follow to decide whether to use NUTS too?

  2. From Appendix A.1 I gather that the NRMSE (as computed for in 3.1 and elsewhere) is applied to posterior samples directly rather than to posterior means? For the former it is not a strictly proper scoring rule [3], because it can be gamed by being overcontracted around the posterior mean. This renders it ill-suited for using in comparisons on its own in principle. Since you are giving also SBC results this would be mitigated, but nonetheless I'd advise being careful and always defining (N)RMSE as the mean accuracy in natural normalized units.

  3. In the main text accuracy metrics are given in significant detail (good) resolved for budget, inference target type and metric, yet to show so much need to be aggregated over datasets. Consequently, the evidence establishes average accuracy at gold-standard level, but does not answer whether individual datasets' inference can be trusted to the same degree. Am I reading this correctly? And can you clarify this in the paper better or investigate further to give conclusive evidence on individual inference accuracy.

  4. Misspecification stress tests: It would be illustrative to have one extreme case of misspecification where we see a difference between NUTS and metabeta and it needs a lot to convince me that there is in fact no difference at all on the individual dataset level, since there are no guarantees for for-outside-trainset generalization of NNs. Ideally, you could add such a negative result to acknowledge that the generalization error is not zero (e.g. increase the value of one predictor from a typical value, say zero, all the way until you see degradation that differs in NUTS and metabeta) OR give an hypothesis why something like this does categorically not exist. It is good enough if you have great generalization, but perfect generalization seems strange.

[3] Gneiting, T., & Raftery, A. E. (2007). Strictly Proper Scoring Rules, Prediction, and Estimation. Journal of the American Statistical Association, 102(477), 359–378. https://doi.org/10.1198/016214506000001437

5 comments
AuthorsOpenReview ↗

Part 1/2: Revised positioning and answers to Q1 and Q2

Thank you for the thorough review. We are glad you see MetaBeta (MB) as a "milestone on the ABI research program" and its novel contributions as "significant on their own". We have substantially updated the positioning, analyzed per-dataset agreement with NUTS, and extended our stress tests according to your proposal. All changes are marked green in the revised PDF.

Positioning

the boundary between related work and novel contributions [...] neglects two [...] publications in the literature, which together arguably make the core idea of metabeta when applied to GLMMs

You are right, [1] already conditions on a prior at test time. [2] already refines draws with importance sampling, or starts many short HMC chains from them. We have updated the draft to acknowledge this:

  • Abstract and introduction no longer claim that prior amortization is new.
  • Related work gained a new paragraph on prior-conditioned NPE that cites SA-ABI [1]. The MCMC paragraph now cites the ABW [2].

The repositioning against the two papers in short: MetaBeta's novelty lies in its scope and how it reaches exactness. For SA-ABI, test-time inputs are the power-scaling exponent for a fixed prior, or the index of a pre-specified prior. For the hierarchical models in their Exp. 3, SA-ABI returns model probabilities, not parameter posteriors. ABW trains one network per model, reweights its draws by importance sampling and reaches exactness through HMC. MetaBeta takes a model structure and a full prior specification (families and hyperparameters) as test-time inputs and returns a full GLMM posterior. Its flow lives inside an IMH kernel, which gives asymptotically exact draws at amortized speed.

Does the revised positioning resolve the over-claiming you pointed out?

Q1

Why is IMH the default, not NUTS warm-starting, is it mainly speed? What is the workflow you propose people follow [...]?

Yes, it's mainly the speed at NUTS accuracy on our benchmarks. Median time per dataset over the 6,144 oracle datasets is 0.15 s for MB (flow + IMH), 18 s for MB+NUTS and 86 s for NUTS. Only default MB makes parallelized model comparison (Sec. 3.4), prior sensitivity analysis (Sec. 3.5), and simulation-based power analysis (App. D.3) practical.

The workflow we propose is now in Sec. 2.3 (and App. B.4):

  1. No time pressure or few analyses: run MB+NUTS directly.
  2. Otherwise: run MB.
  3. If flagged: where the safeguards fire (IMH acceptance < 0.1, ESS < 150 or PSIS-k^\hat k > 0.7), our code warns the user and recommends MB+NUTS.

Q2

From Appendix A.1 I gather that the NRMSE (as computed for in 3.1 and elsewhere) is applied to posterior samples directly rather than to posterior means?

Our NRMSE compares the true parameters with posterior means. We made this more explicit in App. A.1. NRMSE only measures point accuracy, so we evaluate performance on many other axes to get the full picture (coverage, SBC, predictive accuracy, agreement with NUTS in posterior location, scale and shape).

References

AuthorsOpenReview ↗

Part 2/2: Answers to Q3 and Q4

Q3

[...] the evidence establishes average accuracy at gold-standard level, but does not answer whether individual datasets' inference can be trusted to the same degree. Am I reading this correctly? And can you clarify this in the paper better or investigate further to give conclusive evidence on individual inference accuracy.

You're right. We report medians ±\pm MAD over datasets, which gives more of an average picture. We now added a worst-case analysis, which rates each dataset by its biggest disagreement with NUTS over the GLMM parameters (App. C.4).

MB strays from NUTS on 101 of 11,072 datasets (0.9%), and the safeguards catch almost all of them:

test setsdatasetsMB off NUTSunflagged
Gaussian, oracle1,85400
Gaussian, real1,946142
Bernoulli, oracle1,978102
Bernoulli, real1,98630
Poisson, oracle1,912310
Poisson, real1,396434

Q4

It would be illustrative to have one extreme case of misspecification where we see a difference between NUTS and metabeta [...]. Ideally, you could add such a negative result to acknowledge that the generalization error is not zero (e.g. increase the value of one predictor from a typical value, say zero, all the way until you see degradation that differs in NUTS and metabeta) [...].

Thank you for the suggestion. Generalization should fail where the raw flow misses large parts of the true posterior. Then IMH has little to work with and its ESS crashes. To catch this, we adapted the worst-case analysis from Q3.

We first increased the value of one predictor up to |x| = 50 (as you proposed). MB disagreed with NUTS on only 1 of 1,056 datasets. We then checked if MB fails to generalize to oversized datasets (up to 3x larger than its training range, App. E.7). It disagreed with NUTS on 0 of 648 datasets. Finally, we checked how MB handles extreme priors (App. E.3). Prior scales beyond the training range are largely unproblematic, but extreme prior locations and highly skeptical prior scales break the model (percentages pooled over families):

  • Prior scales at 1.11.1 or 2×2\times the trained maximum: MB disagrees with NUTS on 2% of datasets.
  • Prior means shifted by 55 prior SDs: MB disagrees with NUTS on 20% of datasets.
  • Prior scales at 0.5×0.5\times the trained minimum: MB disagrees with NUTS on 49% of datasets.

The safeguards flagged all datasets on which MB disagreed with NUTS (so none failed silently). Sec. 3.6 now closes with: "Overall, MetaBeta proved to be largely resilient against the stress we applied. The most notable disagreement with NUTS arose under extreme priors, far outside the training range (Appendix E.3). The safeguards reliably flagged such cases (Appendix C.4)."


Thank you again for the thoughtful review and the stress-test idea, which resulted in a clearer picture of MB's limits. We are happy to dig deeper into any of the points above.

AuthorsOpenReview ↗

Follow-up on our revision

Quick follow-up, since discussion closes on Wednesday (Nov 18). You wrote that you "expect to change [your] review favorably" if we rework the positioning of MetaBeta. We no longer present prior amortization as new (Abstract, Sec. 1), and in Related Work we now credit SA-ABI and ABW and state where MetaBeta differs. Does this resolve the over-claiming? If so, we would be very grateful if your score reflected this (as well as the per-dataset analysis and the negative result you requested, App. C.4 and E.3). If not, we are happy to fix any line you point out before the deadline.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Overview of the revision

We thank all three reviewers for their constructive feedback. Reviewers see MetaBeta as "a significant milestone on the ABI research program" whose "experiments are compelling" (yUwz), "a great advance" that "resolves a major open problem in amortization" (YHwa), and "a comprehensive solution to the proposed problem, with extensive experiments that support its effectiveness" (u7oa).

We mark every requested change in the updated PDF using a green font:

  • yUwz: positioning (abstract, Sec. 1), workflow after a flag (Sec. 2.3, App. B.4), NRMSE (App. A.1), per-dataset agreement (Sec. 3.1, 3.2, App. C.4), stress tests (Sec. 3.6, App. E.3, E.7).
  • YHwa: pointers (Sec. 2, Table 1, Fig. 2), random-effect design (Sec. 2.1, App. G.2), trained prior ranges (Sec. 3.6, App. E.2, App. B.4), out-of-support priors (App. E.3).
  • u7oa: practical need (Sec. 1, App. G.1), model-comparison runtime (App. D.1), power analysis (App. D.3), sentence splits (abstract).

Three other changes since submission (not colored):

  • We added Pathfinder as a reference method next to ADVI, so MetaBeta now runs against both standard fast alternatives to NUTS (Sec. 3, App. A.3).
  • We made the NUTS reference more stable. PyMC's default budget converges on only 41-67% of oracle datasets, so each NUTS fit now escalates through two larger budgets until R^\hat R <= 1.01, ESS >= 400 and 0 divergences. The largest budget converges on 84-98% and serves as the reference (App. A.3). NUTS runtimes are based on the first converged budget per fit.
  • We validated model comparison for Bernoulli and Poisson models at every size (App. D.1).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.