- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper investigates a variational objective that interpolates between the expected individual-model log-loss and the log-loss of the ensemble predictive mixture. The authors derive observation-dependent responsibilities for different ensemble members, an optimal parameter distribution, and a local stability criterion. They further conduct practical experiments in which the entropy regularization term is omitted, and the method is applied to a LoRA ensemble with a fixed number of adapters and a shared backbone. The results are qualitatively consistent with the proposed specialization mechanism: ensemble disagreement increases for larger values of , while accuracy and the mean OR-Bench refusal rate remain broadly stable.
Critical strengths
- The paper is well motivated, and the theoretical derivations support the central intuition that the proposed objective can induce specialization among ensemble members. Compared with approaches that rely on explicit gating mechanisms, the proposed method does not require a separate gating network, since member specialization is induced through the responsibility scores.
- The empirical results are qualitatively consistent with the proposed mechanism: disagreement increases for larger values of , while task accuracy remains relatively stable.
Critical weaknesses
-
The empirical evidence for the practical benefits of the method is limited. The two evaluated tasks primarily demonstrate that disagreement increases for larger values of , whereas task performance remains approximately unchanged. This shows that the method can induce diversity, but it does not yet establish whether this additional diversity is practically useful. It would strengthen the paper to demonstrate that the increased ensemble diversity translates into improvements in downstream uncertainty-related metrics, such as error detection AUROC, predictive accuracy, or uncertainty calibration metrics such as ECE or Brier score, for example on MMLU.
-
The relationship to prior work requires further discussion. The proposed objective appears closely related to the objective introduced in [1], up to differences in scaling and notation. In the empirical implementation, the entropy term is omitted, and the resulting method is also closely related to LoRA-Ensemble [2]. The paper would benefit from a more explicit discussion of the similarities and differences with these approaches. Additional empirical comparisons against these methods would further clarify the novelty and practical contribution of the proposed approach.
References
[1] Futami et al. (2022). Predictive Variational Bayesian Inference as Risk-Seeking Optimization. AISTATS.
[2] Mühlematter et al. (2026). LoRA-Ensemble: Efficient Uncertainty Modelling for Self-Attention Networks. TMLR.
Questions for authors
- Does increased disagreement improve calibration (e.g., ECE), error detection (e.g., AUROC), or downstream task performance in general compared with the independent LoRA-Ensemble?
- Which theoretical predictions do the experiments validate after dropping the parameter-distribution entropy term?
4 comments
Thanks for the response and the new results.
Could you give the exact definition of the mutual-information score used for error detection in Table 1? Equation (18) uses each adapter’s probability for the correct option, so I’d like to confirm whether the score in Table 1 can be computed without knowing the correct answer.
Thank you for your reply!
As you mentioned the definition of the mutual-information score is provided in Eq. (18). We clarify it here for convenience, let the test dataset of question prompts and their corresponding correct answers .
Let and . For a uniformly sampled test question and particle , correctness is Bernoulli with parameter , giving
\left[h(\bar q_j)-\frac1M\sum_m h(q_{mj})\right],$$ where $h(q)=-q\log_2q-(1-q)\log_2(1-q)$, with arguments clipped to $[10^{-12},1-10^{-12}]$. So in order to compute the conditional particle-correctness MI metric reported in Figure 2, the true answer is needed. However, the uncertainty score used for error detection in Table 1 does not require the true answer. Ground-truth labels are used only subsequently to evaluate how well this score distinguishes incorrect from correct predictions. In particular, let $p_m(\cdot\mid x_j)$ denote the distribution of member $m$ over the four answer options (a softmax over the A-D logits), and let $\bar p_j = M^{-1}\sum_m p_m(\cdot\mid x_j)$ be the ensemble distribution. For error detection, each test question is scored by the mutual information between the member index and the predicted answer $$u_j=\widehat I(J;Y\mid X=x_j)=H(\bar p_j)-\frac1M\sum_{m=1}^M H\big(p_m(\cdot\mid x_j)\big),\qquad H(p)=-\sum_{k=1}^{4}p_k\log p_k .$$ This score uses only the members predictions, not the true answer $y_j$. With the ensemble prediction $\hat y_j=\arg\max_k \bar p_j(k)$, denote errors $\mathcal E=\\{j:\hat y_j\neq y_j\\}$ and correct predictions $\mathcal C=\\{j:\hat y_j=y_j\\}$, the error-detection AUROC is $$\mathrm{AUROC}=\frac{1}{|\mathcal E|\,|\mathcal C|}\sum_{i\in\mathcal E}\sum_{l\in\mathcal C}\Big(\mathbb 1[u_i>u_l]+\tfrac12\,\mathbb 1[u_i=u_l]\Big).$$ Hope this clarifies your question!Thank you for the clarification. I’m raising my score by one point.
Thank you for the thoughtful review. We appreciate your positive comments and address your concerns and questions below.
Thanks for raising this point, we explore this in our MMLU setting.
Table 1. Downstream uncertainty metrics of the ensemble on MMLU (Nemotron-3-8B, test set, 14042 questions). Δ is the change relative to deep ensembles, with 95% bootstrap CIs in brackets. Bold marks a CI that excludes zero.
Table 2. Adapter diversity on questions the ensemble gets wrong versus right (same runs). Temperature for ECE is fitted on the MMLU validation split.
These results show that the additional diversity is associated with improved uncertainty estimates in this setting. Error-detection AUROC increases, while ECE after temperature scaling decreases, with both changes supported by bootstrap intervals excluding zero. Disagreement increases more strongly on incorrect predictions, widening the wrong–correct gap from 0.171 to 0.274. These findings support benefits for error detection and post-hoc calibration beyond increased disagreement alone. We have added this analysis in App. D.2 of the updated version of the paper.
As mentioned in the paper, the KL regularisation term is infinite for the empirical ensemble QM when the prior π0 has a Lebesgue density. We therefore use the surrogate specified in Section 4, retaining the expected prior potential and dropping the negative entropy term
Lα(Q)=∑n=1Nℓα(Q;xn,yn)+λEQ[−logπ0].
This objective remains convex under linear interpolation of probability measures Q, although strong convexity and uniqueness are no longer guaranteed. However, the stability result in Corollary 1 explicitly considers the prior-potential surrogate without the negative entropy term and gives a local condition under which spreading the parameter distribution lowers the objective. Our experiments test this specialisation mechanism.
We agree that our per-observation loss coincides with the entropic-risk loss of Futami et al. (2022) after rescaling. However, we arrive at this loss from a different objective. Futami et al. start from risk-seeking optimisation of a parametrised variational posterior qϕ, with mean-field Gaussians among the families they analyse. We arrive at the same loss by interpolating between the geometric and arithmetic means of the member likelihoods. These endpoints correspond to averaging the individual log-losses and scoring the predictive mixture, respectively. This same distinction applies at α=1, between predictive Bayes in a chosen variational family and the log-score PrO posterior of McLatchie et al. The former fits qϕ, while the latter defines its target over admissible laws Q. We take the latter view throughout 0<α≤1 providing a general framework and derive a self-consistency equation for the optimal law in Proposition 1.
Mühlematter et al. (2026) share a frozen backbone across distinct low-rank adapters. Our independent LoRA baseline uses this construction and appears as the deep ensemble in Figs. 2 and 4.