acceptodds
Under review as a conference paper at ICLR 2027

Large Language Bayes Is Not Reparameterisation-Invariant

Abstract

Large Language Bayes (LLB) answers an informal modelling question by sampling candidate probabilistic programs from a language model, running approximate inference on each, and averaging them with weights proportional to an exponentiated evidence bound. We show that this weighting depends on how a model is written. The log marginal likelihood is invariant to reparameterisation; the evidence bound is not. On eight schools the centered and non-centered programs are the same measure to , yet their weights differ by ; importance weighting reduces this only to , and reproducing the inference LLB actually runs, a full-covariance Gaussian matched to the posterior moments, still leaves on eight schools and in dimensions. Across likelihood families, dimensions and funnel severities the discrepancy reaches and reverses sign, so no single writing is uniformly preferable. It inverts Bayes factors against eight natural competitors, and we bound the model-posterior error it induces, and that of any downstream target, by , tightly. Across programs from six language models the parameterisation written ranges from to centered and is stable within a model. Detecting equivalent programs statistically can falsely merge genuinely different models at practical sample budgets; verifying reparameterisations we generate ourselves cannot, and closes the window.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Bayesian model selection, variational inference, probabilistic programming, large language models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_JptpOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the parameterisation dependence of evidence-bound-based model weighting in Large Language Bayes (LLB). Using centered and non-centered hierarchical models, it demonstrates that equivalent representations can receive different scores, analyses the consequences for model comparison, examines LLMs’ parameterisation preferences, and proposes a mitigation that searches for tighter bounds using templates for equivalent transformations.

Critical strengths
  1. Section 4.2 uses a full-covariance Gaussian proposal matched to posterior moments, showing that the parameterisation dependence is not restricted to mean-field VI. Computing these moments through numerical integration also helps isolate the effect from posterior sampling error.
  2. The reversal in the direction of the discrepancy across settings demonstrates that always choosing a non-centered representation is not a universally effective solution. Reference evidence values and controlled competing models make the scoring discrepancies and ranking reversals straightforward to examine.
Critical weaknesses
  1. The underlying mechanism is well studied, and the novelty and significance of the contribution are limited.

    The dependence of approximate inference on parameterisation, particularly the differences between centered and non-centered hierarchical models, has long been studied (e.g., Papaspiliopoulos et al., 2007). The effect of evidence approximation errors on model weights is also not a new observation: Section 5.3 and Appendix C.4 of the original LLB paper explicitly analyse model-dependent evidence-bound errors.

    The contribution therefore primarily consists of applying and quantifying an established mechanism in LLB, rather than identifying a new failure mechanism. The main total variation bound is also a specialisation of an existing result. Such a diagnosis can still be valuable, but it needs to establish sufficiently important new insights or practical consequences. The paper currently neither convincingly demonstrates the effect on LLB’s final outputs nor establishes its empirical significance beyond this particular workflow. My concern is not the focused scope itself, but that the limited theoretical increment is not sufficiently complemented by consequential empirical findings.

  2. The experiments are not sufficiently aligned with the LLB workflow being criticised, and some claims exceed the evidence.

    The paper does not extend the evaluation to the original LLB tasks and generated models, relying primarily on predefined hierarchical models. The 360 programs in Section 5 are classified by parameterisation without running the full inference pipeline. Moreover, its NumPyro and minimal-prompt setup differs from the original LLB setup, which uses Stan and worked examples. These experiments establish representation preferences under the tested conditions, but do not directly establish how frequently or substantially actual LLB outputs are affected.

    Section 4.2 does reproduce the proposal construction used in LLB, which is an important strength, but it remains a controlled evaluation of an inference component. The headline numbers in the abstract and conclusion largely come from the mean-field setting, potentially overstating how directly they characterise the original workflow. Furthermore, parameterisation frequencies are not failure rates, particularly given the demonstrated reversals in the direction of the discrepancy. Ranking reversals and worst-case error bounds also do not replace measurements of changes to the final model-averaged posterior.

    A similar mismatch appears in the claim that the repair “closes the window”: the tested window is substantially reduced, while improvements to actual LLB outputs under a matched computational budget have not been demonstrated. These issues make the practical claims stronger than the available evidence.

Questions for authors
  1. Relative to existing work on parameterisation and the original LLB analysis of evidence-bound errors, what is the most important new insight provided by this paper? How does it go beyond applying and quantifying a known mechanism in a particular workflow?

  2. Could you evaluate a limited subset of the original LLB tasks or generated programs, holding the statistical hypotheses fixed and changing only equivalent parameterisations, and report the resulting changes in normalised model weights and target posteriors? If repair results are available, please also report the additional computational cost. This would be more informative about the contribution’s significance than further controlled parameter sweeps or representation-frequency statistics.

Reviewer_XT4cOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies a reparameterisation-invariance issue in Large Language Bayes (LLB). It shows that mathematically equivalent probabilistic programs can receive different model weights due to differences in their approximate evidence bounds. The authors demonstrate how this can reverse model rankings and study the parameterisation preferences of different LLMs. They also propose a template-based method that generates equivalent parameterisations and selects the tightest evidence bound.

Critical strengths
  1. Clear problem formulation: The paper highlights an important issue in LLB. The mathematical analysis clearly explains how different representations of the same probabilistic model can lead to inconsistent model weights.

  2. Useful experimental analysis: The experiments show that the issue persists across different model configurations and inference settings. The study of LLM-generated programs also provides useful evidence about parameterisation preferences across different LLMs.

Critical weaknesses
  1. Limited novelty: The main observation builds on the well-known sensitivity of variational inference to parameterisation. While the paper demonstrates its implications for LLB, the underlying issue is not new. The proposed repair also relies on existing reparameterisation ideas and selecting tighter evidence bounds. The methodological novelty therefore seems limited.

  2. Limited practical evaluation: The repair is mainly demonstrated on the eight-schools example and is not evaluated in a complete LLB pipeline. It also relies on predefined templates, which limits its applicability to more complex probabilistic programs. Furthermore, as noted in Appendix B, tighter individual evidence bounds do not necessarily improve the overall model posterior. A broader evaluation would help establish the practical benefits of the approach.

Questions for authors
  1. What is the main methodological contribution beyond applying known reparameterisation techniques to LLB?

  2. Could the authors evaluate the repair in a complete LLB pipeline and show whether it improves the final model posterior?

  3. How well does the repair generalize beyond location-scale hierarchical models, and what is the additional computational cost?

  4. The related work could also discuss recent research on the reliability of LLM-generated probabilistic programs, such as RefineStat (ICLR 2026) and Likelihood Hacking in Probabilistic Program Synthesis (UAI 2026). These works address different issues but would provide useful context for the paper.

Reviewer_BnHvOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

LLB uses variational evidence estimates as model weights, but different parameterizations lead to different behaviors under approximate inference. This paper identifies the reparameterization problem for LLBs and proposes a fix where centered and non-centered parameterizations are both used.

Critical strengths
  • The paper identifies a limitation of LLB that has not been explicitly studied.
  • The proposed solution is simple and effective in their limited empirical settings
Critical weaknesses
  • The writing is poor and unfocused. For instance, the four listed contributions do not read as immediate results and instead have numbers and experiment results that are very difficult to interpret. Many sentences throughout the entire paper feel awkward or unfinished.
  • The paper spends a majority of the paper explaining established relationships between parameterization and approximate inference, and the authors even acknowledge that "centered and non-centered hierarchies behaving differently under approximate inference is long established." When the authors investigate the LLM component in Section 5, they run one experiment which categorizes the parameterization of the LLMs, but do not run the full LLB pipeline, leaving no end-to-end demonstration of the issues they proposed.
Questions for authors
  • Have you tried running this end-to-end? Can you demonstrate how these difference impact the final model weights?
  • Does the proposed solution generalize beyond the specific eight-schools example?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.