acceptodds
Under review as a conference paper at ICLR 2027

Random-Set Large Language Models

Abstract

Large Language Models (LLMs) are known to produce very high-quality texts and responses to our queries. But how much can we trust this generated text? In this paper, we study the problem of uncertainty quantification in LLMs. We propose a novel Random-Set Large Language Model (RS-LLM) approach which predicts finite random sets (belief functions) over the token space, rather than probability vectors as in classical LLMs. In order to allow so efficiently, we also present a methodology based on hierarchical clustering to extract and use a budget of “focal” subsets of tokens upon which the belief prediction is defined, rather than using all possible collections of tokens, making the method scalable yet effective. RS-LLMs encode the epistemic uncertainty induced in their generation process by the size and diversity of its training set via the size of the credal sets associated with the predicted belief functions. The proposed approach is evaluated on multiple datasets using Llama2-7b, Llama3.1-8b, Mistral-7b and Phi-2 models and compared against existing uncertainty methods, and is shown to outperform the standard model and existing uncertainty methods in both datasets in terms of correctness of answer, while also showing potential in estimating the second-level uncertainty of its predictions and empowering the ability to detect when it is hallucinating.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Uncertainty Quantification, Random Sets

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_mpcYOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work proposes a modification to existing LLM architectures based on random set theories, outputting belief functions instead of token probabilities. A hierarchical clustering mechanism is developed to decide the set size for good scalability. The new architecture is expected to provide better uncertainty quantifications and prediction performances based on empirical studies.

Critical strengths

The main idea of incorporating random sets and belief function prediction into LLMs is interesting and inspiring. The scalable hierarchical clustering also makes the approach practical to implement.

Critical weaknesses
  1. The tasks and base models tested are still limited: only QA tasks are tested, and the base models are a bit outdated. The generalizability of the approach to other task types and more SOTA models is not clear.
  2. The uncertainty quantification is only measured qualitatively (Figure 4). A more thorough quantitative comparison of uncertainty quantification performances among different baselines are needed.
Questions for authors
  1. Can you provide a training cost comparison before and after applying the random-set modifications to LLM architectures? The additional cost of the new architecture is unclear.
  2. The uncertainty estimation of the approach is response-wise instead of prompt-wise, right? Response-wise uncertainty/confidence has unique advantages, and it would be interesting to compare or discuss its connection to other SOTA response-wise uncertainty/confidence metrics, e.g., semantic density [1].

[1] “Semantic Density: Uncertainty Quantification for Large Language Models through Confidence Measurement in Semantic Space”, NeurIPS 2024.

Reviewer_GZeoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper tries to propose a new LLM generation diagram, RS-LLM, by modeling the output of LLMs as finite random sets (belief functions) to make it possible to directly reflect the uncertainty from models’ outputs. RS-LLM has a relatively solid theoretical ground and has the nature to directly represent the model’s uncertainty without multiple sampling. However, the empirical evidence and experimental coverage are relatively limited.

Critical strengths
  1. The paper's writing is good and easy to follow.
  2. The method is clearly presented and has a solid theoretical ground.
Critical weaknesses
  1. RS-LLM needs additional training/fine-tuning with a new training objective. The models used in the paper are relatively small, and the tasks used to evaluate are easy. My main concern is that It’s not clear if RS-LLM can scale up well.
  2. The scope is restricted to QA tasks or multiple-choice tasks.
  3. For uncertainty estimation, the experiments lack some important baselines. For example, Semantic Entropy, Predictive Entropy, and other uncertainty estimation methods for LLMs.
Questions for authors
  1. Can the authors add more uncertainty estimation baselines?
  2. Can the authors test their method on relatively new models and tasks? For example, Qwen and MMLU Pro.
Reviewer_dqFvOpenReview ↗
Rating
1
Confidence
2
Informativeness
3
Presentation
2
Soundness
1
Read review
Paper summary

The paper adapts Random-Set Neural Networks (RS-NNs) to LLMs. The main obstacle it addresses is the size of the output space. LLM vocabularies range from tens of thousands to over 100K tokens (32K–128K for the models used here), which makes naive belief functions over the full power set infeasible. The proposed approach covers only a small subset of the power set: all singletons plus a comparatively small number of non-singleton focal sets (K = 8,000). These focal sets are obtained by agglomerative clustering of the tokens by similarity in the embedding space. The model (LoRA adapters plus a new output layer) is then trained to output belief functions over this budget with a custom loss: binary cross-entropy on the belief values, plus penalties on negative masses and on total mass above 1. Uncertainty is quantified in two ways: the entropy of the pignistic distribution, which is also used to generate the predictions, and the width of the credal set. Experiments with four small LLMs (Phi-2, Llama2-7B, Mistral-7B and Llama-3.1-8B) on four QA datasets (CoQA and OBQA for all four models; ARC-E and BoolQ for Llama2 only) suggest gains in both accuracy and uncertainty quantification.

Critical strengths
  1. Motivation and positioning. The paper is well motivated and clearly states its relation to prior work on RS-NNs.
  2. Apparent, if modest, novelty. The paper appears to be the first to adapt random-set prediction to LLMs. Its new elements are the definition of focal sets by clustering token embeddings and the token-level application to generation. The loss and uncertainty measures follow RS-NNs (Manchingal et al., 2025), apart from how masses are recovered from beliefs (Eq. 10).

References:

• Manchingal, S. K., Mubashar, M., Wang, K., Shariatmadar, K. and Cuzzolin, F. Random-set neural networks. ICLR 2025.

Critical weaknesses
  1. Training targets carry no set-valued information. The targets are labelled with "full certainty" (§4.1): Bel(A) = 1 if the observed next token is in A and 0 otherwise, so all of the target's mass sits on the observed token. It is unclear how a model trained only on such targets learns to output more expressive belief functions, with mass on non-singleton sets.

  2. The budgeting captures uncertainty in wording rather than meaning. The focal sets added by the budgeting procedure appear to be clusters of semantically close tokens. High belief in such a set combined with low belief in its singletons then corresponds to uncertainty of expression, not uncertainty of meaning. This goes against the predominant view in well-known LLM uncertainty quantification methods (Farquhar et al., 2024; Fadeeva et al., 2024; Duan et al., 2024), which move uncertainty from the level of particular expressions to the level of distinct meanings. A more detailed justification of the budgeting approach is needed.

  3. Claims of significant gains lack significance analysis. The contributions and the text describe the gains as significant (Contribution 4, §5.2, Conclusions), but no significance analysis is performed beyond the ± values over 5 runs in Tab. 1. §5.2 states that RS-Llama2 "significantly outperforms all the other methods" on the basis of Tab. 2, which reports no variance at all. In Tab. 3, the ± values are of the same magnitude as the values themselves, and the paper does not say what they measure (spread across examples or across runs); the AUROCs come with no confidence intervals. RS-Llama2 also has the highest AUROC on only one of the two datasets (BoolQ, 86.34); on OBQA, BLoB is higher (92.45 vs. 90.23).

References:

• Duan, J. et al. Shifting attention to relevance: towards the predictive uncertainty quantification of free-form large language models. ACL 2024.

• Fadeeva, E. et al. Fact-checking the output of large language models via token-level uncertainty quantification. Findings of ACL 2024.

• Farquhar, S., Kossen, J., Kuhn, L. and Gal, Y. Detecting hallucinations in large language models using semantic entropy. Nature 630, 625–630, 2024.

Questions for authors
  1. Is the standard baseline in Tab. 1 (also the "Llama2" row of Tab. 2) fine-tuned on each dataset with ordinary next-token cross-entropy, using the same LoRA configuration and schedule, or is it the base model without fine-tuning? §5.2 says all models are trained with "the exact same regimen", but the rows carry base-model names.
  2. Please define CC and IC in Tab. 3 (presumably correct and incorrect context) and state what the ± values denote, and distinction between which cases does the AUROC report in this table?
  3. Which uncertainty score is used for the standard model, BLoB and LoRA ensembles in Tab. 3 and Fig. 4? For RS-LLMs the paper states pignistic entropy (Tab. 3) and credal width (Tab. 10), but Tab. 3's caption names pignistic entropy, which is defined only for RS-LLM.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.