- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes a method for improving the uncertainty calibration of large language models (LLMs) by increasing ensemble diversity through evidentially motivated sample weighting of varying strengths. The method trains multiple LoRA adapters on a shared pretrained backbone. Token logits are converted into evidential belief and vacuity scores, which are then used to determine sample weights of different strengths for each ensemble member.
At inference time, responses are sampled from the ensemble and grouped according to semantic equivalence. The resulting answer frequencies are used to determine both the final prediction and its associated confidence. The paper reports improvements in calibration and downstream task performance compared with established baselines.
Critical strengths
-
Important practical problem. Uncertainty estimation after fine-tuning foundation models is an important practical problem for safe deployment. While prior work has focused on efficient ensemble architectures or stochastic decoding from a single model, this work proposes a plausible approach for improving calibration by explicitly increasing output diversity.
-
Improvement over a strong baseline. The proposed method improves over a vanilla ensemble, which is a challenging and relevant baseline for uncertainty quantification.
Critical weaknesses
-
Incremental novelty of the proposed method. The proposed approach appears closely related to existing techniques. In particular, given its similarity to focal loss, the authors should include a focal-loss ensemble baseline using the same hyperparameter-selection protocol as the proposed method. Furthermore, the idea of increasing ensemble diversity through different weightings of training samples is conceptually similar to well-established and simpler techniques such as bagging or training on random subsets of the data. Direct comparisons with these approaches would help clarify the added value of the proposed method. Additionally, the hyperparameters used in the individual experiments, as well as the procedure used to select them, are not described clearly enough. In particular, the choice of (\gamma) appears to have a substantial effect on performance according to Table 4. It therefore remains unclear how sensitive the proposed method is to hyperparameter choices across datasets and models, and whether these hyperparameters were selected using a held-out validation set.
-
Missing directly relevant ensemble literature. The paper does not sufficiently discuss closely related work on LoRA-based ensembles. In particular, the proposed construction is related to prior LoRA ensemble methods [1], which also aim to increase ensemble diversity and can exhibit particularly strong diversity for smaller LoRA ranks relative to fully fine-tuned ensembles. A clearer discussion and empirical comparison would help position the contribution.
Reference [1] Mühlematter et al. LoRA-Ensemble: Efficient Uncertainty Modelling for Self-Attention Networks. TMLR, 2026
Questions for authors
-
Is the same set of gamma hyperparameters used across all datasets and model sizes? How were these values selected or tuned?
-
How does the proposed method compare with bagging and focal-loss ensembles under equal hyperparameter-tuning budgets?
-
How does the ensemble diversity of the proposed method compare with that of competing approaches, and how strongly does diversity correlate with downstream task performance and calibration? For example, the authors could consider disagreement scores, Jensen–Shannon divergence between predictive distributions, or differences between ensemble members' learned parameters or weights.
-
How does the method compare with a vanilla ensemble when using a larger number of ensemble members, such as 6 or 8, if computationally feasible?