- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies the parameterisation dependence of evidence-bound-based model weighting in Large Language Bayes (LLB). Using centered and non-centered hierarchical models, it demonstrates that equivalent representations can receive different scores, analyses the consequences for model comparison, examines LLMs’ parameterisation preferences, and proposes a mitigation that searches for tighter bounds using templates for equivalent transformations.
Critical strengths
- Section 4.2 uses a full-covariance Gaussian proposal matched to posterior moments, showing that the parameterisation dependence is not restricted to mean-field VI. Computing these moments through numerical integration also helps isolate the effect from posterior sampling error.
- The reversal in the direction of the discrepancy across settings demonstrates that always choosing a non-centered representation is not a universally effective solution. Reference evidence values and controlled competing models make the scoring discrepancies and ranking reversals straightforward to examine.
Critical weaknesses
-
The underlying mechanism is well studied, and the novelty and significance of the contribution are limited.
The dependence of approximate inference on parameterisation, particularly the differences between centered and non-centered hierarchical models, has long been studied (e.g., Papaspiliopoulos et al., 2007). The effect of evidence approximation errors on model weights is also not a new observation: Section 5.3 and Appendix C.4 of the original LLB paper explicitly analyse model-dependent evidence-bound errors.
The contribution therefore primarily consists of applying and quantifying an established mechanism in LLB, rather than identifying a new failure mechanism. The main total variation bound is also a specialisation of an existing result. Such a diagnosis can still be valuable, but it needs to establish sufficiently important new insights or practical consequences. The paper currently neither convincingly demonstrates the effect on LLB’s final outputs nor establishes its empirical significance beyond this particular workflow. My concern is not the focused scope itself, but that the limited theoretical increment is not sufficiently complemented by consequential empirical findings.
-
The experiments are not sufficiently aligned with the LLB workflow being criticised, and some claims exceed the evidence.
The paper does not extend the evaluation to the original LLB tasks and generated models, relying primarily on predefined hierarchical models. The 360 programs in Section 5 are classified by parameterisation without running the full inference pipeline. Moreover, its NumPyro and minimal-prompt setup differs from the original LLB setup, which uses Stan and worked examples. These experiments establish representation preferences under the tested conditions, but do not directly establish how frequently or substantially actual LLB outputs are affected.
Section 4.2 does reproduce the proposal construction used in LLB, which is an important strength, but it remains a controlled evaluation of an inference component. The headline numbers in the abstract and conclusion largely come from the mean-field setting, potentially overstating how directly they characterise the original workflow. Furthermore, parameterisation frequencies are not failure rates, particularly given the demonstrated reversals in the direction of the discrepancy. Ranking reversals and worst-case error bounds also do not replace measurements of changes to the final model-averaged posterior.
A similar mismatch appears in the claim that the repair “closes the window”: the tested window is substantially reduced, while improvements to actual LLB outputs under a matched computational budget have not been demonstrated. These issues make the practical claims stronger than the available evidence.
Questions for authors
-
Relative to existing work on parameterisation and the original LLB analysis of evidence-bound errors, what is the most important new insight provided by this paper? How does it go beyond applying and quantifying a known mechanism in a particular workflow?
-
Could you evaluate a limited subset of the original LLB tasks or generated programs, holding the statistical hypotheses fixed and changing only equivalent parameterisations, and report the resulting changes in normalised model weights and target posteriors? If repair results are available, please also report the additional computational cost. This would be more informative about the contribution’s significance than further controlled parameter sweeps or representation-frequency statistics.