acceptodds
Under review as a conference paper at ICLR 2027

Soft Specialists: -Rényi Ensembles for Uncertainty-Aware LLM Post-Training

Abstract

Existing training approaches for large language models learn a single set of parameters, based on large volumes of data, which is typically heterogeneous, conflicting and often outright contradictory. As a result, the model is forced to compress conflicting goals, and inherent uncertainties into a single, averaged pattern of behaviour. We propose an -Rényi variational framework for learning distributions over post-training parameters, offering an uncertainty-aware alternative to deep ensemble approaches. The resulting variational objective interpolates between classical variational Bayes and predictively oriented posterior learning, balancing between globally plausible individual models against systems of complementary specialists. We identify local stability criteria, demonstrating how model misspecification can make non-degenerate posterior spread locally favourable, manifesting contradictory or conflicting data as epistemic uncertainty. We apply our framework to LLM post-training, learning an ensemble of LoRA adapters attached to a shared, frozen base model, providing a scalable training procedure for both supervised fine-tuning and preference optimisation. Our approach enables training examples to be softly routed across ensemble members, promoting model specialisation and providing actionable uncertainty estimates across different tasks.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
LLM post-training, uncertainty quantification, low-rank adaptation (LoRA), ensemble learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_edsbOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper investigates a variational objective that interpolates between the expected individual-model log-loss and the log-loss of the ensemble predictive mixture. The authors derive observation-dependent responsibilities for different ensemble members, an optimal parameter distribution, and a local stability criterion. They further conduct practical experiments in which the entropy regularization term is omitted, and the method is applied to a LoRA ensemble with a fixed number of adapters and a shared backbone. The results are qualitatively consistent with the proposed specialization mechanism: ensemble disagreement increases for larger values of α\alpha, while accuracy and the mean OR-Bench refusal rate remain broadly stable.

Critical strengths
  • The paper is well motivated, and the theoretical derivations support the central intuition that the proposed objective can induce specialization among ensemble members. Compared with approaches that rely on explicit gating mechanisms, the proposed method does not require a separate gating network, since member specialization is induced through the responsibility scores.
  • The empirical results are qualitatively consistent with the proposed mechanism: disagreement increases for larger values of α\alpha, while task accuracy remains relatively stable.
Critical weaknesses
  • The empirical evidence for the practical benefits of the method is limited. The two evaluated tasks primarily demonstrate that disagreement increases for larger values of α\alpha, whereas task performance remains approximately unchanged. This shows that the method can induce diversity, but it does not yet establish whether this additional diversity is practically useful. It would strengthen the paper to demonstrate that the increased ensemble diversity translates into improvements in downstream uncertainty-related metrics, such as error detection AUROC, predictive accuracy, or uncertainty calibration metrics such as ECE or Brier score, for example on MMLU.

  • The relationship to prior work requires further discussion. The proposed objective appears closely related to the objective introduced in [1], up to differences in scaling and notation. In the empirical implementation, the entropy term is omitted, and the resulting method is also closely related to LoRA-Ensemble [2]. The paper would benefit from a more explicit discussion of the similarities and differences with these approaches. Additional empirical comparisons against these methods would further clarify the novelty and practical contribution of the proposed approach.

References

[1] Futami et al. (2022). Predictive Variational Bayesian Inference as Risk-Seeking Optimization. AISTATS.

[2] Mühlematter et al. (2026). LoRA-Ensemble: Efficient Uncertainty Modelling for Self-Attention Networks. TMLR.

Questions for authors
  1. Does increased disagreement improve calibration (e.g., ECE), error detection (e.g., AUROC), or downstream task performance in general compared with the independent LoRA-Ensemble?
  2. Which theoretical predictions do the experiments validate after dropping the parameter-distribution entropy term?
4 comments

Thank you for the thoughtful review. We appreciate your positive comments and address your concerns and questions below.

Does increased disagreement improve calibration (e.g., ECE), error detection (e.g., AUROC), or downstream task performance in general compared with the independent LoRA-Ensemble?

Thanks for raising this point, we explore this in our MMLU setting.

Table 1. Downstream uncertainty metrics of the ensemble on MMLU (Nemotron-3-8B, test set, 14042 questions). Δ\Delta is the change relative to deep ensembles, with 95% bootstrap CIs in brackets. Bold marks a CI that excludes zero.

MetricDeep ensembleα\alpha = 0.25α\alpha = 0.5α\alpha = 0.75α\alpha = 1
Error-detection AUROC, mutual-information score ↑0.5940.6040.6180.6340.647
  Δ\Delta vs deep ensemble–+0.010 [+0.006, +0.014]+0.025 [+0.019, +0.030]+0.040 [+0.033, +0.047]+0.053 [+0.044, +0.063]
ECE after temperature scaling (%) ↓3.553.303.332.551.64
   Δ\Delta vs deep ensemble (pp)–−0.26 [−0.50, +0.16]−0.23 [−0.58, +0.36]−1.00 [−1.38, −0.43]−1.92 [−2.39, −0.93]

Table 2. Adapter diversity on questions the ensemble gets wrong versus right (same runs). Temperature for ECE is fitted on the MMLU validation split.

MetricDeep ensembleα\alpha = 0.25α\alpha = 0.5α\alpha = 0.75α\alpha = 1
Adapter disagreement, wrong / correct answers0.344 / 0.1730.356 / 0.1750.383 / 0.1810.418 / 0.2040.505 / 0.231
  Gap (wrong − correct)+0.171+0.181+0.202+0.214+0.274
Errors where at least one adapter was correct (%) ↑29.329.731.834.643.5
Errors where all adapters gave the same wrong answer (%) ↓42.340.136.932.321.4

These results show that the additional diversity is associated with improved uncertainty estimates in this setting. Error-detection AUROC increases, while ECE after temperature scaling decreases, with both changes supported by bootstrap intervals excluding zero. Disagreement increases more strongly on incorrect predictions, widening the wrong–correct gap from 0.1710.171 to 0.2740.274. These findings support benefits for error detection and post-hoc calibration beyond increased disagreement alone. We have added this analysis in App. D.2 of the updated version of the paper.

Which theoretical predictions do the experiments validate after dropping the parameter-distribution entropy term?

As mentioned in the paper, the KL regularisation term is infinite for the empirical ensemble QMQ_M when the prior π0\pi_0 has a Lebesgue density. We therefore use the surrogate specified in Section 4, retaining the expected prior potential and dropping the negative entropy term

L~α(Q)=∑n=1Nℓα(Q;xn,yn)+λ EQ[−log⁡π0].\widetilde{\mathcal{L}}_\alpha(Q)=\sum_{n=1}^N \ell_\alpha(Q;x_n,y_n)+\lambda\,\mathbb E_Q[-\log\pi_0].

This objective remains convex under linear interpolation of probability measures QQ, although strong convexity and uniqueness are no longer guaranteed. However, the stability result in Corollary 1 explicitly considers the prior-potential surrogate without the negative entropy term and gives a local condition under which spreading the parameter distribution lowers the objective. Our experiments test this specialisation mechanism.

The relationship to prior work requires further discussion.

We agree that our per-observation loss coincides with the entropic-risk loss of Futami et al. (2022) after rescaling. However, we arrive at this loss from a different objective. Futami et al. start from risk-seeking optimisation of a parametrised variational posterior qϕq_\phi, with mean-field Gaussians among the families they analyse. We arrive at the same loss by interpolating between the geometric and arithmetic means of the member likelihoods. These endpoints correspond to averaging the individual log-losses and scoring the predictive mixture, respectively. This same distinction applies at α=1\alpha=1, between predictive Bayes in a chosen variational family and the log-score PrO posterior of McLatchie et al. The former fits qϕq_\phi, while the latter defines its target over admissible laws QQ. We take the latter view throughout 0<α≤10<\alpha\leq1 providing a general framework and derive a self-consistency equation for the optimal law in Proposition 1.

Mühlematter et al. (2026) share a frozen backbone across distinct low-rank adapters. Our independent LoRA baseline uses this construction and appears as the deep ensemble in Figs. 2 and 4.

Thanks for the response and the new results.

Could you give the exact definition of the mutual-information score used for error detection in Table 1? Equation (18) uses each adapter’s probability for the correct option, so I’d like to confirm whether the score in Table 1 can be computed without knowing the correct answer.

Thank you for your reply!

As you mentioned the definition of the mutual-information score is provided in Eq. (18). We clarify it here for convenience, let the test dataset (xj,yj)j=1N\\{(x_j, y_j)\\}_{j=1}^N of question prompts xjx_j and their corresponding correct answers yjy_j.

Let qmj=pm(yj∣xj)q_{mj}=p_m(y_j\mid x_j) and qˉj=M−1∑mqmj\bar q_j=M^{-1}\sum_m q_{mj}. For a uniformly sampled test question XX and particle JJ, correctness CC is Bernoulli with parameter qmjq_{mj}, giving

\left[h(\bar q_j)-\frac1M\sum_m h(q_{mj})\right],$$ where $h(q)=-q\log_2q-(1-q)\log_2(1-q)$, with arguments clipped to $[10^{-12},1-10^{-12}]$. So in order to compute the conditional particle-correctness MI metric reported in Figure 2, the true answer is needed. However, the uncertainty score used for error detection in Table 1 does not require the true answer. Ground-truth labels are used only subsequently to evaluate how well this score distinguishes incorrect from correct predictions. In particular, let $p_m(\cdot\mid x_j)$ denote the distribution of member $m$ over the four answer options (a softmax over the A-D logits), and let $\bar p_j = M^{-1}\sum_m p_m(\cdot\mid x_j)$ be the ensemble distribution. For error detection, each test question is scored by the mutual information between the member index and the predicted answer $$u_j=\widehat I(J;Y\mid X=x_j)=H(\bar p_j)-\frac1M\sum_{m=1}^M H\big(p_m(\cdot\mid x_j)\big),\qquad H(p)=-\sum_{k=1}^{4}p_k\log p_k .$$ This score uses only the members predictions, not the true answer $y_j$. With the ensemble prediction $\hat y_j=\arg\max_k \bar p_j(k)$, denote errors $\mathcal E=\\{j:\hat y_j\neq y_j\\}$ and correct predictions $\mathcal C=\\{j:\hat y_j=y_j\\}$, the error-detection AUROC is $$\mathrm{AUROC}=\frac{1}{|\mathcal E|\,|\mathcal C|}\sum_{i\in\mathcal E}\sum_{l\in\mathcal C}\Big(\mathbb 1[u_i>u_l]+\tfrac12\,\mathbb 1[u_i=u_l]\Big).$$ Hope this clarifies your question!

Thank you for the clarification. I’m raising my score by one point.

Reviewer_usXrOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes an α\alpha-Rényi variational framework for learning a distribution over LLM post-training parameters. Each observation is scored by the α\alpha-power mean of its likelihood, so the objective interpolates between generalized variational Bayes as α→0\alpha \to 0 and the predictively oriented posterior at α=1\alpha = 1.

On the theoretical side, the authors show that the objective is convex in QQ and characterize its minimizer through a self-consistent fixed-point equation. Their main result is a local stability criterion based on the matrix V−αJV - \alpha J, which identifies a critical value αcrit\alpha_{\rm crit} beyond which posterior spread becomes favorable under model misspecification.

In practice, the distribution is approximated by an ensemble of LoRA adapters on a shared frozen backbone. Training reduces to responsibility-weighted SFT or DPO updates, where each example is softly routed toward the adapters that currently explain it best.

Experiments on the Massive Multitask Language Understanding benchmark MMLU with four models show that larger α\alpha increases disagreement between adapters while ensemble accuracy stays stable. On the over-refusal benchmark OR-Bench after DPO training with three models, larger α\alpha mainly increases refusal disagreement on hard benign prompts, while mean refusal rates remain nearly unchanged.

Critical strengths

The main theoretical strength of the paper is the local stability analysis in Proposition 3. Rather than only stating that model misspecification can make posterior spread beneficial, the authors give a precise and interpretable criterion based on the matrix V−αJV - \alpha J. This criterion tells us when spread becomes favorable through the threshold αcrit\alpha_{\rm crit}, in which parameter directions it occurs, and, as illustrated nicely in Example 1, in which regions of the input space the resulting uncertainty appears. I also appreciate that the criterion is invariant under reparametrization and directly connects the classical sandwich mismatch V≠JV \neq J to ensemble diversity. This is a clean and insightful result that, to my knowledge, is not available in prior work on predictively oriented or risk-seeking posteriors.

The second strength is the application to LLM post-training. As far as I know, this is the first work to bring predictively oriented posterior learning to the scale of LLMs, and the extension to preference optimization is a natural and welcome contribution. The resulting procedure is simple to implement, since it only modifies the training loss and requires no separate gating network, and it remains efficient because all adapters share a frozen backbone. The refusal experiment is also interesting from an alignment perspective. Disagreement between adapters increases mainly on hard benign prompts while the average refusal behavior stays almost unchanged, which suggests a promising direction for identifying ambiguous requests that may deserve further review.

Critical weaknesses
  1. Limited novelty of the objective.

    • The same α\alpha-power loss with KL regularization already appears in black-box α\alpha-divergence minimization [1, 2], and was studied as a risk-seeking predictive objective in [3].
    • Since α=1\alpha = 1 recovers the predictively oriented posterior [4], the method is essentially PrO with an α\alpha-power.
    • Interpolating between predictive and parameter-oriented objectives was also explored in predictive variational inference [5], which is not discussed.
  2. Theory largely follows the PrO paper [4].

    • Proposition 1 is an α\alpha-generalization of the PrO fixed-point characterization, and existence of the minimizer is assumed rather than proven.
    • Proposition 2 and the claim that misspecification favors spread are weaker, population-level versions of known PrO results.
    • Proposition 3 appears to be the only genuinely new theoretical result.
  3. Unclear benefit of α<1\alpha < 1.

    • In all experiments, α=1\alpha = 1 performs best or equally well.
    • PrO guarantees such as convergence and predictive dominance are not extended to α<1\alpha < 1.
    • No principled rule for choosing α\alpha is given.
  4. Proposition 3 is disconnected from the experiments.

    • V−αJV - \alpha J and αcrit\alpha_{\rm crit} are never estimated in the LLM setting.
    • The analysis assumes infinitesimal spread with full KL regularization, while the implementation uses a few particles, a prior surrogate, and deterministic AdamW.
    • It would help to check whether the predicted threshold matches when specialization actually emerges.

References

[1] Hernandez-Lobato, Jose, et al. "Black-box alpha divergence minimization." International conference on machine learning. PMLR, 2016.

[2] Li, Yingzhen, and Yarin Gal. "Dropout inference in Bayesian neural networks with alpha-divergences." International conference on machine learning. PMLR, 2017.

[3] Futami, Futoshi, et al. "Predictive variational Bayesian inference as risk-seeking optimization." International Conference on Artificial Intelligence and Statistics. PMLR, 2022.

[4] McLatchie, Yann, et al. "Predictively oriented posteriors." arXiv preprint arXiv:2510.01915 (2025).

[5] Lai, Jinlin, Antonio Linero, and Yuling Yao. "Predictive variational inference: Learn the predictively optimal posterior distribution." arXiv preprint arXiv:2410.14843 (2024).

Questions for authors
  1. Guarantees for α<1\alpha < 1.

    • For α=1\alpha = 1, PrO is known to converge to the predictively optimal model average and to predictively dominate Bayes and Gibbs posteriors [1].
    • Do similar guarantees hold for the α\alpha-posterior over general QQ when α∈(0,1)\alpha \in (0, 1)? Prior consistency results such as [2] are limited to parametric variational families.
    • More importantly, are there conditions under which α<1\alpha < 1 is provably better than α=1\alpha = 1? This would give a clear justification for the proposed interpolation.
  2. Existence of the minimizer in Proposition 1.

    • Proposition 1 assumes that a minimizer with a strictly positive density exists, while existence and uniqueness are proven for PrO [1].
    • Could the authors establish existence for α∈(0,1)\alpha \in (0, 1), or clarify under which conditions it may fail?
  3. Empirical benefit of intermediate α\alpha.

    • In all reported results, α=1\alpha = 1 performs best or equally well.
    • Could the authors report metrics where intermediate α\alpha may help, such as the accuracy or response quality of individual adapters, calibration measured by ECE or NLL, and selective prediction?
  4. Empirical validation of Proposition 3.

    • Could the authors estimate V−αJV - \alpha J in the LoRA parameter space, for example via Hessian-vector products and Lanczos iterations?
    • Does the predicted αcrit\alpha_{\rm crit} match the value of α\alpha at which disagreement between adapters actually starts to increase?
  5. Missing details and baselines.

    • Please report the value of λ\lambda used in each experiment.
    • Section 4 mentions a smoothed empirical-density estimate of the KL term, but only the prior surrogate is used. How does the smoothed version perform?
    • How do training and inference costs compare with a single LoRA adapter and with deep ensembles, and how sensitive are the results to the number of adapters MM?
    • Comparisons with multiple choice learning [3, 4], mixture-of-LoRA-experts approaches, and PrO computed with Langevin noise as in [1] would help position the method.
    • Please report wall-clock time, FLOPs, and peak GPU memory for training and inference, compared with a single LoRA adapter and with deep ensembles, and discuss sensitivity to the number of adapters MM.

References

[1] McLatchie, Yann, et al. "Predictively oriented posteriors." arXiv preprint arXiv:2510.01915 (2025).

[2] Futami, Futoshi, et al. "Predictive variational Bayesian inference as risk-seeking optimization." International Conference on Artificial Intelligence and Statistics. PMLR, 2022.

[3] Guzman-Rivera, Abner, Dhruv Batra, and Pushmeet Kohli. "Multiple choice learning: Learning to produce multiple structured outputs." Advances in neural information processing systems 25 (2012).

[4] Lee, Stefan, et al. "Stochastic multiple choice learning for training diverse deep ensembles." Advances in Neural Information Processing Systems 29 (2016).

9 comments

Thank you for the insightful review. We appreciate your positive comments and address your questions below.

Q1. Guarantees and advantages of α<1\alpha<1

Additional guarantees. For α<1\alpha<1 we provide a result showing that the α\alpha-posterior over general QQ dominates Bayes and Gibbs posteriors in the misspecified setting and converge to the predictively optimal α\alpha-power mixture. See Theorem 1 (Appendix B.8) in the updated version of the paper.

Theoretical advantage of α<1\alpha<1. At the population level, Proposition 2 shows that in the well-specified setting, every minimiser of the unregularised population risk for 0<α<10<\alpha<1 is supported on the exact-fit set, so its constituent models individually recover the true distribution. For α=1\alpha=1, only the predictive mixture must recover the truth, potentially combining individually misspecified models. Under identifiability, the former therefore uniquely selects δθ⋆\delta_{\theta^\star}.

Additionally at the finite-sample level, in the well-specified regime, we show on a two-model space that the finite sample predictive risk of the α\alpha-posterior dominates over the predictive posterior. So while both procedures can converge to the optimal predictive distribution, the strict advantage concerns their finite-sample expected risks and convergence rates, see Theorem 2 (Appendix B.8) in the updated version.

Q2. Existence of the minimizer in Proposition 1

Thanks for raising this point. It is indeed possible to show existence of the minimizer in Proposition 1, we have updated the proposition accordingly (see updates marked in red).

Q3. Empirical benefits of intermediate α\alpha

To complement the theoretical results from Q1, we provide empirical evidence of the advantage of α<1\alpha<1 in a simple well-specified setting where it is easier to isolate the different effects (see new App. D.5)

We train ensembles of M=10M=10 members on the Two Moons classification task. Each member is a two-layer ReLU MLP defining a predictive distribution pθi(y∣x)p_{\theta_i}(y\mid x) over the two labels. We train the ensemble for a grid of α\alpha values in [0,1][0,1] and regularisation weights λ∈0,10−3,10−2,10−1\lambda\in\\{0,10^{-3},10^{-2},10^{-1}\\}, repeating each configuration over 5050 seeds. We report the ensemble test accuracy and the mean test accuracy across its individual members.

Figure 7 shows that intermediate α\alpha improves mean ensemble test accuracy over α=1\alpha=1 for each regularisation weight considered. The highest observed mean ensemble accuracies occur at α≈0.6\alpha\approx0.6-0.850.85, exceeding those at α=1\alpha=1 by 1.191.19 percentage points for λ=0.01\lambda=0.01. Mean individual-member accuracy generally decreases as α\alpha increases, with a decline of 2.862.86-3.253.25 percentage points from α=0\alpha=0 to α=1\alpha=1 across the regularisation settings. Thus, intermediate values improve ensemble accuracy relative to α=1\alpha=1 while retaining stronger individual members.

Q4. Empirical validation of Proposition 3

We have added App. D.4, which empirically validates the predicted critical value in a controlled LoRA regression experiment based on Example 1 where we can compute αcritical\alpha_{\mathrm{critical}} in closed form.

We estimate α^critical\widehat{\alpha}_{\mathrm{critical}} from the Hessian and empirical Fisher in the adapter parameter space, these estimates concentrate around the theoretical value 25/7325/73 as the sample size increases (Figure 5). Starting from slightly perturbed copies of a fitted adapter, the observed transition from decreasing to increasing predictive spread closely matches the estimated threshold α^critical=0.3502\widehat{\alpha}_{\mathrm{critical}}=0.3502. When using our α\alpha-Renyi objective for values α<α^critical\alpha<\widehat{\alpha}_{\mathrm{critical}} posterior spread collapses, while for larger values we observe in Figure 6 an exponential increase in the spread.

Extending this quantitative validation to LLMs remains future work, as obtaining reliable estimates α^critical\widehat{\alpha}_{\mathrm{critical}} in their high-dimensional, overparameterised setting requires additional care.

Q5. Missing details

  • Regarding the parameter λ\lambda we use 1e-4 in the MMLU example and 1e-3 in the DPO example. We have added them in the updated version.

  • We have removed the reference to the smoothed empirical-density estimate of the KL term since we introduced it as a possible alternative approach but in the end only used the prior surrogate since it is the common regularisation used in practice when training neural networks.

  • We report below the training and inference cost for Nemotron-3-8B on MMLU, using one NVIDIA A100-80GB. Each ensemble has M = 8 rank-4 LoRA adapters. Training is run for 5 epochs on the 1502 MMLU validation questions. Inference covers the full MMLU test set (14042 questions).

Cost per configuration

ConfigurationTraining time (h)Training peak memory (GB)Inference time (min)Inference peak memory (GB)Latency (ms/question)
Frozen base model––3.318.214.2
Single LoRA adapter (estimated)0.2317.93.418.214.5
Deep ensemble0.4420.413.721.458.6
α\alpha-Rényi ensemble (averaged of α\alpha = 0.25, 0.5, 0.75, 1)0.4420.413.821.459.0

These measurements show that the α\alpha-Rényi objective introduces negligible additional overhead relative to an independent LoRA ensemble of the same size. Standard optimisations for LoRA ensembles, including parallelisation and sharding, are also applicable to our method and could further improve its computational efficiency.

  • Related approaches

Multiple choice learning encourages specialisation by assigning each example to its lowest-loss member which our loss ℓα\ell_\alpha recovers as α→∞\alpha \to \infty. This approach has been identified in the literature to suffer from member collapse and overconfidence due to the hard assigments [6, 7] which we overcome inducing soft responsabilities controlled by α\alpha.

Mixture-of-LoRA-experts approaches learn an input-dependent gating network that is used at test time, whereas our responsibilities require no gate and arise directly from the loss through the observed response during training. These are therefore different classes of predictor, and a direct comparison would conflate the objective with the routing architecture.

Langevin-based PrO targets the same predictive objective at α=1\alpha = 1 but accounts for the entropy term through diffusion, which requires discretisation and convergence diagnostics that are difficult to assess at LLM scale.

Comparison with previous literature

We agree that our per-observation loss coincides with the entropic-risk loss of Futami et al. (2022) [3] after rescaling. However, we arrive at this loss from a different objective. Futami et al. start from risk-seeking optimisation of a parametrised variational posterior qϕq_\phi, with mean-field Gaussians among the families they analyse. We arrive at the same loss by interpolating between the geometric and arithmetic means of the member likelihoods. These endpoints correspond to averaging the individual log-losses and scoring the predictive mixture, respectively. This same distinction applies at α=1\alpha=1, between predictive Bayes in a chosen variational family and the log-score PrO posterior of McLatchie et al. The former fits qϕq_\phi, while the latter defines its target over admissible laws QQ. We take the latter view throughout 0<α≤10<\alpha\leq1 providing a general framework and derive a self-consistency equation for the optimal law in Proposition 1.

Regarding connections to [1, 2], we mentioned them in the introduction but we will discuss them in more detail. The original BB-α\alpha objective of Hernández-Lobato et al. [1] uses tied site approximations and includes a density-ratio correction inside each likelihood expectation, so it is generally different from our objective. However, Li and Gal [2] derive an approximation whose power-loss-plus-KL functional coincides with ours at unit regularisation weight, and implement it using dropout variational distributions. We acknowledge this overlap. Our contribution develops the power-mean interpretation and analysis over general admissible laws QQ, including the optimal-law characterisation and the local stability criterion, and applies this framework to LLM post-training.

Predictive variational inference [5] is also closely related: its log-score objective with prior-KL regularisation coincides with our α=1\alpha=1 endpoint when restricted to its variational family. It additionally interpolates between predictive and parameter-oriented inference through regularisation towards the Bayesian posterior. This yields a linear combination of predictive-mixture and expected individual log-losses, together with prior regularisation. Our power-mean construction follows a different interpolation between these endpoints. We will clarify both this shared motivation and the distinction between the two objectives.

[1] Hernandez-Lobato, Jose, et al. "Black-box alpha divergence minimization." International conference on machine learning. PMLR, 2016.

[2] Li, Yingzhen, and Yarin Gal. "Dropout inference in Bayesian neural networks with alpha-divergences." International conference on machine learning. PMLR, 2017.

[3] Futami, Futoshi, et al. "Predictive variational Bayesian inference as risk-seeking optimization." International Conference on Artificial Intelligence and Statistics. PMLR, 2022.

[4] McLatchie, Yann, et al. "Predictively oriented posteriors." arXiv preprint arXiv:2510.01915 (2025).

[5] Lai, Jinlin, Antonio Linero, and Yuling Yao. "Predictive variational inference: Learn the predictively optimal posterior distribution." arXiv preprint arXiv:2410.14843 (2024).

[6] Mike Brodie, Chris Tensmeyer, Wes Ackerman, and Tony Martinez. Alpha model domination in multiple choice learning. In IEEE International Conference on Machine Learning and Applications, ICMLA, 2018.

[7] Victor Letzelter, Mathieu Fontaine, Mickael Chen, Patrick Perez, Slim Essid, and Gaël Richard. Resilient Multiple Choice Learning: A learned scoring scheme with application to audio scene analysis. In Thirty-Seventh Conference on Neural Information Processing Systems, November 2023.

Reviewer_xtPuOpenReview ↗
Rating
2
Confidence
1
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper shows uncertainty aware post training approach for LLMs using Reny divergence formulation. The loss function is introduced in Eq. (2) and Section 3 develops mathematical theory around the loss function. The theoretical contribution is generic to the application that follows in Section 4 on Lora fine-tuning. This paper shows an application of the theory to fine-tuning on two benchmarks in the experiment section. From the reportings, the accuracy remains the same over 4 alpha values, and it is higher than the frozen base few percent and matches to deep ensemble mean.

Critical strengths

Theoretical formulations around the proposed objective functions.

Critical weaknesses

From the experiment, the effect of new approach seems limited and demonstration on small pilot experiments.

Questions for authors

Besides the theoretical formulation, Algorithm 1 is the procedure that adopts the proposed approach. Could you explain the benefit or improvement by the proposed approach?

2 comments

We thank the reviewer for their time. We answer your questions and concerns below:

Could you explain the benefit or improvement by the proposed approach?

Our approach connects the classical Bayesian and predictively oriented posterior viewpoints. The predictively oriented posterior (which coincides with our α=1\alpha=1 case) can improve prediction under misspecification. In the well-specified setting, its population minimisers recover the true predictive distribution, but need not concentrate on individually correct models. At the other endpoint, classical Bayes can recover the true parameter under standard consistency conditions, but fails under misspecification. Our objective, parameterised by α\alpha, provides a middle ground between these viewpoints. We prove that, in the well-specified setting, every minimiser of the unregularised population risk for 0<α<10<\alpha<1 is supported on models that individually recover the true data-generating distribution (Proposition 2). Under misspecification, we identify a critical value above which posterior spread becomes favourable and correctly approximates the posterior. Additionally, choosing α<1\alpha<1 can mitigate imbalances in learning across ensemble members, consistent with the improved individual-member accuracy observed in Appendix D.5.

Additionally, when approximating the posterior with particles (like LoRA adapters in the language model case) we observe that disagreement among the different members increases as α\alpha grows (see Figures 2 and 3 in the paper) especially in questions that the ensemble mixture gets wrong disagreement between the members of the ensemble is larger than in questions that the ensemble gets right (see Table 2 for the MMLU setting) or in highly ambiguous questions like in the safety OR-Bench setting (see Figure 3). We show in Table 1 how disagreement can translate into downstream uncertainty-related metrics which might be highly useful for decision making.

We would like to emphasise that for the main results, our metric of interest is not accuracy. We supply the accuracy purely to showcase that model performance is not hindered under our loss, in comparison to other methods such as BayesLoRA, while providing other benefits in terms of uncertainty quantification.

Regarding our experiments, we test our approach on finetuning and direct preference optimisation tasks on models ranging from 1.5B to 8B parameters which we consider high dimensional practical settings.

Thanks very much for your reply.

As the authors pointed out, accuracy is not the primary target of the improvement and the benefits are coming from uncertainty quantification. The theory suggests that the extension using α\alpha-interpolation should help when the post-training data is a mixture of consistent and inconsistent preferences. Then, α\alpha may keep the adapters accurate on the consistent part, by Proposition 2. Proposition 3 shows that the inconsistent portion makes the posterior spread locally acceptable when α\alpha is greater than αcrit\alpha_{crit}. It seems like the current paper is testing two extremes of α\alpha. In Figure3, the refusal rate in each group is essentially same accross 3 different α\alpha values. This might suggest that α\alpha does not change the ensemble's helpfulness-harmlessness trade-off. Table 1 shows that AUROC and ECE show improvement over 4 α\alpha values, and it is the most effective at α=1\alpha=1. The experiment in the paper doesn't have a case of using α\alpha brings clear benefit. During post-training, α\alpha is a hyperparameter that needs to be decided, and if we choose extreme values (0 or 1), then the proposed theory supports the motivation of using intermediate values, but it doesn't translate into improvement in practice. The main theoretical contribution of this paper is extending the baseline papers with α\alpha-Reny ensembles. Most of the results are generalizations of the baseline theory. There isn't a strong connection to LLM post-training.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.