acceptodds
Under review as a conference paper at ICLR 2027

Credal Large Language Models for Semantic Commitment under Uncertainty

Abstract

Large language models (LLMs) often produce fluent but incorrect answers with unwarranted confidence. A central limitation is that standard LLMs represent uncertainty through a single predictive distribution, conflating epistemic ignorance with genuine ambiguity. We introduce Credal Large Language Models (CLLMs): an ensemble of LoRA adapters induces a credal set whose lower and upper probabilities expose the spread of plausible predictive distributions rather than collapsing to a single softmax output. From this representation, we derive a single commitment rule: the model commits to an answer only when its lower probability exceeds the upper probability of every alternative, and otherwise returns the set of answers that no plausible predictor rules out. We apply this commitment rule at two depths: Credal Token Commitment (CTC) applies it to answer tokens from one ensemble forward pass, which decides constrained answers without any generation; for open-ended answers, credal decoding extends a partial answer only when no completed answer dominates it, so that the completions produced are those the plausible predictors license, and Credal Semantic Commitment (CSC) applies the rule to their meaning clusters. We evaluate CLLMs with Gemma-2-9B, Llama-3.1-8B and Qwen2.5-7B on OpenBookQA, CoQA, TriviaQA and ARC-Challenge. On multiple choice, CTC commits on 73-91% of questions at 89-98% accuracy, returns sets of 1.1-1.5 options containing the gold one on 89-98%, and its intervals contain the observed accuracy in 24 of 30 confidence bins without calibration; corrupted context lowers commitment from 87-92% to 65-71%, and on Gemma the credal bound detects corruption better than every baseline. On open-ended QA, CLLM outperforms semantic entropy and Laplace-LoRA at a fixed coverage by up to 19% and 9.5% absolute accuracy on CoQA and TriviaQA with context, for every backbone.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Uncertainty Quantification, Credal sets

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_nDEUOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper applies the concept of credal sets and the decision-theoretic interval-dominance rule to obtain better answers and a more accurate confidence score in LLMs. As a basis, the authors propose using the output probability distributions over answers from different LoRA models and differently than in a classical Bayesian setting do not average them, but rather represent them as a credal set. The major contribution is the credal decoding algorithm, which decodes multiple sequences in parallel and selects which to pursue based on the interval-dominance rule. Secondly, the authors apply the interval-dominance rule both at the token level and at the semantic sequence level to map a credal set to a prediction set and, next, retrieve a confidence assigned to it. In experiments across multiple datasets, the retrieved confidence outperforms sampling-based methods and confidence scores derived from LoRA ensembles or Bayesian LoRA methods on multiple objectives, ranging from separating correct from incorrect predictions (AUROC) to calibration metrics such as ECE and NLL.

Critical strengths

Key Strength 1

Sound Theory & Good Translation into a Practical Method

Credal Set-based approaches have recently received more attention in the UQ literature. Using interval dominance to make decisions is an interesting approach that the authors translated well into an LLM setting. The idea of using interval dominance at the token or semantic level would have been interesting but small-scale, so I see the key contribution in the credal decoding algorithm, which I find well thought through and with nice properties.

Consequence for assessment: The method is theoretically motivated, and the theory is well translated into practice.

Not a key strength, but nonetheless, I want to highlight this:

Especially compared to other papers in my batch, this paper was significantly better written and, in my opinion, meets the bar for an A* conference (manuscript-wise).

Critical weaknesses

Key Weakness 1

Concern about short sequence bias in credal decoding

In Credal Decoding (3.3), the updates are an addition of log probabilities without length normalization. This inherently favors shorter sequences. The algorithm at every step (once at least one completed sequence has been generated) checks if a partial sequence (which is at least the same length?) has an upper mass greater than the lowest mass of the completed sequence. Naturally, as the partial sequence grows longer, its probability decreases; hence, at some point, it seems quite likely that every partial sequence will be pruned simply to the fact that a completed sequence is shorter. It seems that evidence for this is itself presented in Experiments ((iv) L410).

A simple fix would be to length-normalize the sequence probabilities, which is commonly used in decoding strategies. Unfortunately, I think that this would break the math regarding the unexplored mass, as, to my understanding, the formulation in equation (4) assumes that the whole distribution over all sequences sums to 1. This would unfortunately no longer be given with length normalization.

Consequence for assessment: As of now, it seems to me that the proposed method is unsuitable for generating longer sequences, and a straightforward fix would break the math. This limits the method's applicability to very short tasks.

Key Weakness 2

Ensemble baseline comparison

As L315/316 states, “ensemble methods rank the same answer, the CSC representative”, i.e., the prediction to which the confidence estimate is provided belongs to the candidate provided by the credal decoding method. Naturally, the credal decoding confidence estimate (the largest lower probability) is tailored to exactly this prediction, whereas this is not the case for the ensemble methods. As such, I am questioning whether this is an unfair advantage here. Moreover, as it seems (Appendix D.3), the LoRA Ensemble uses only the first token distribution to build predictive entropy and mutual information. Especially for multi-token predictions, this seems like a severe handicap that could be fixed by using a distribution over semantic classes, as in the credal decoding method.

Consequence for assessment: As of now, I am not confident in the comparison of numbers in Tab1 and Tab2 between CLLM and LoRA Ensembles.

Questions for authors

Q1: Key Weakness 1

Could you please comment on the short-sequence bias and (if feasible) present a small subset of results using length-normalized probabilities?

Q2: Key Weakness 2

Please evaluate the numbers in Tab1 and Tab2 for LoRA Ensembles and CLLM, both when greedy and when using, e.g., the MAP estimate of the LoRA predictive posterior distribution. Moreover, please comment on the implementation of the ensemble and the problem that they only use the first token distribution. Should this be the case, an extension to use the semantic distribution would be helpful.

Other Questions:

Q3: Compute: If correctly understood at every step, the algorithm has B partial sequences and M adapters. Hence, one step runs BxM forward passes (According to experiment setup D.4, this is then 8x5 = 40 forward passes). This seems like substantial compute and is not equivalent to M model passes (as stated in the limitations - this is only the case for 1 token prediction). Could you please clarify this and potentially revise the limitations?

Q4: Accuracy of Credal Decoding: For me, the core part of the paper is the credal decoding method, which can find different sequences than, e.g., beam search or greedy search. The paper's framing, however, emphasizes the uncertainty-confidence aspect. Do you have evidence that if

Q5: What is the sense of recording a truncation? This was mentioned in the algorithm but then never again.

Suggestions to the authors

  • Overall, I am missing a bit of the red thread (beyond the theory) for the main goal. Is it creating better answers through credal decoding? Is it creating a good confidence score that outperforms existing ones (UQ literature)? Is it providing set-based prediction? All of these topics are discussed, and I think the paper would benefit from narrowing the story further and placing additional material in the appendix.
  • Similar to the point above, I think one could bring CTC and CSC CLLMs together under the umbrella of interval dominance on credal sets. Then one could emphasize the section on credal decoding more.
  • Minor Things:
    • Tables on page 7 are very close to the text (overall, I feel the main text could be made a bit more concise)
    • Figure 3: “The rule decides where a tuned threshold would, and withdraws when the evidence is corrupted” → Caption seems to be missing something
    • Appendix E8 → blue color of text

While I am currently leaning toward a weak reject, I am very much open to raising my score if my points are addressed or clarified.

10 comments

Dear Reviewer,

Thank you for your detailed review, for your interest in credal decoding, and for being open to revisiting your score. We ran the experiments you suggested, and answer each point below.

W1, Q1:

In Credal Decoding (3.3), the updates are an addition of log probabilities without length normalization. This inherently favors shorter sequences. The algorithm at every step (once at least one completed sequence has been generated) checks if a partial sequence (which is at least the same length?) has an upper mass greater than the lowest mass of the completed sequence. Naturally, as the partial sequence grows longer, its probability decreases; hence, at some point, it seems quite likely that every partial sequence will be pruned simply to the fact that a completed sequence is shorter.

We thank you for the careful reading of §3.3. Joint probabilities do favour shorter sequences, but pruning does not remove a continuation for being long: it removes it only when, as a whole answer, it is less probable under every adapter than one already completed.

Each partial sequence carries every adapter's log-probability of its tokens (§3.3, L226-227). Whatever its length, it is compared with the largest lower mass q∗q^* of the completed sequences so far (L232-233), with no length condition, and pruned only when it is dominated as a sequence (L240-242): its upper mass, the most any adapter gives to it and to every completion of it, is below q∗q^*. Every answer it could reach is then less probable than that completed sequence under every adapter. Long completions survive: on the CoQA development split the longest completion per prompt averages 13.1 tokens for Llama and 17.9 for Qwen, and only 1.9% of completions on CoQA with Llama are cut at L (§D.4, L1203-1205; L252).

It seems that evidence for this is itself presented in Experiments ((iv) L410).

L410 of the submission is in the four-option setting, where CTC decides from the first answer token alone, so the pruning of credal decoding plays no part in that result.

It is in §4.2 (iv) (L415 in the revised version), where CTC decides from one ensemble forward pass at the first answer token (L414-415), before any sequence is extended.

A simple fix would be to length-normalize the sequence probabilities, which is commonly used in decoding strategies. Could you please comment on the short-sequence bias and (if feasible) present a small subset of results using length-normalized probabilities? Unfortunately, I think that this would break the math regarding the unexplored mass, as, to my understanding, the formulation in equation (4) assumes that the whole distribution over all sequences sums to 1. This would unfortunately no longer be given with length normalization.

We have run the suggested length normalisation: it keeps the mass accounting of Eq. (4) exact, because the unexplored mass is computed from the joint probabilities of the completed sequences, and it barely changes the decisions on the CoQA development split.

The pruning test compares per-token geometric means; everything else is unchanged.

Table G: Length-normalised pruning. CoQA development split, 500 prompts, M=5, B=8, L=32.

BackbonePruningCSC commit rate (%)Accuracy when committed (%)Gold-in-set (%)Completion with the largest lower probability, committed prompts (tokens)Longest completion per prompt (tokens)
Llama-3.1-8Bjoint probability (paper)40.892.289.24.5513.1
Llama-3.1-8Blength-normalised42.291.590.04.5622.5
Qwen2.5-7Bjoint probability (paper)29.093.888.84.4117.9
Qwen2.5-7Blength-normalised30.094.089.04.4023.0

Length-normalised pruning follows about twice as many completions (Table G), and longer ones, yet CSC commitment changes by 1.0-1.4 points, accuracy when committed by at most 0.7 points, and the completion with the largest lower probability on committed prompts keeps its length (4.55 vs 4.56 tokens on Llama, 4.41 vs 4.40 on Qwen). On these two settings the measured decisions are insensitive to the pruning score. This is now Tab. 20 in §E.7.2 (L1774), and §3.3 states that pruning depends on probability, not on length (L242).

Eq. (4) (L247-252) defines umu_m from the completed sequences only, and whatever the decoder leaves out remains visible through umu_m (L252-254). Length normalisation only changes which partial sequences are followed; completed sequences keep their joint probabilities pm(h∣x)p_m(h \mid x), so um(x)=1−∑hpm(h∣x)u_m(x) = 1 - \sum_h p_m(h \mid x) still holds and the completion set and the unexplored mass sum to 1 for every adapter, for any rule that chooses which continuations to follow. Nothing in the pruning restricts answer length: L=32 is a budget fixed on the development splits (L1203-1205), and B and L do not decide which completions exist (L252-254). Our evaluation covers short-answer QA.

W2, Q2: We thank you for these questions, which we have answered with the requested experiments. Scoring on the CSC representative does not favour CLLM: on its own MAP answer the LoRA Ensemble shows no systematic gain, and we have added the LoRA Ensemble over meanings you suggest.

In Tabs. 1 and 2 every method built on the adapters ranks the CSC representative, so only the score differs (L318-320). We now also score the LoRA Ensemble on its own MAP answer (the meaning with the largest mean-pooled probability) and on its own greedy answer (decoded token by token from the mean-pooled distribution of the five adapters). The MAP answer equals the CSC representative on 86-92% of open-ended prompts for Llama and Gemma and 73-86% for Qwen, and on 94-99.6% of four-option questions, where greedy and MAP coincide. It is less accurate than the CSC representative on TriviaQA with the passage (85, 80 and 75% against 86, 83 and 76%) and level or slightly more accurate on CoQA (75, 79 and 76% against 75, 78 and 74%; Tab. 11, k=1, and Tab. 3). The greedy answer of the base model is the Standard LLM in Tabs. 1 and 2 (§4.1, L308-310; §D.3, L1168-1169).

Table H: The LoRA Ensemble scored on its own answer. The paper's LoRA Ensemble (predictive entropy, five adapters mean-pooled): change when it is scored on its own MAP answer or its own greedy answer instead of the CSC representative, as in Tabs. 1 and 2; all three answers are scored in the same run. On the four-option datasets the greedy answer is the MAP letter.

SettingAUROC against own errors (Tab. 1): change on own MAP answer / own greedy answerAcc@80% (Tab. 2), points: change on own MAP answer / own greedy answer
CoQA, Llama-0.022 / -0.013+0.0 / -5.0
CoQA, Gemma-0.003 / +0.030+1.5 / -4.0
CoQA, Qwen-0.003 / -0.003+1.5 / -1.5
TriviaQA with the passage, Llama-0.009 / -0.001-0.5 / -2.5
TriviaQA with the passage, Gemma-0.002 / +0.007-1.5 / -10.0
TriviaQA with the passage, Qwen+0.011 / +0.004-1.5 / -2.5
Closed-book TriviaQA, Llama-0.004 / -0.014+1.5 / -3.0
Closed-book TriviaQA, Gemma+0.010 / +0.014-1.5 / -7.0
Closed-book TriviaQA, Qwen+0.019 / +0.025+1.0 / -4.5
OpenBookQA, Llama+0.020 / +0.020+1.0 / +1.0
OpenBookQA, Gemma-0.016 / -0.016+0.0 / +0.0
OpenBookQA, Qwen-0.004 / -0.004+0.0 / +0.0
ARC-Challenge, Llama+0.023 / +0.023-1.3 / -1.3
ARC-Challenge, Gemma+0.008 / +0.008+0.0 / +0.0
ARC-Challenge, Qwen-0.008 / -0.008+0.0 / +0.0

On its own MAP answer (Table H), the ensemble's AUROC against its own errors changes by at most 0.023 with predictive entropy (0.029 with mutual information) and its Acc@80% by at most 1.5 points (2.0), and the predictive-entropy AUROC is lower in 9 of the 15 settings and higher in 6. Its corrupted-context AUROC cannot change, because predictive entropy and mutual information do not depend on the answer returned. On its own greedy answer, the ensemble is less accurate than the CSC representative in all nine open-ended settings, by 1-13 points (Q4), its Acc@80% is lower in all nine, by 1.5-10.0 points, and its AUROC moves by at most 0.030. The comparison in Tabs. 1 and 2 is therefore not sensitive to which answer the ensemble is scored on.

The paper's LoRA Ensemble uses the first-token distributions, following the original code base (§D.3, L1175-1178; §4.1, L310-311). We added the extension you propose: the five adapters mean-pooled over the same meanings as CSC, from the same credal decoding run, scored by predictive entropy, mutual information and one minus the probability of its MAP meaning, each on its own MAP answer.

Table I: LoRA Ensemble over meanings. Corrupted-context detection, AUROC; each LoRA Ensemble score is computed on its own MAP answer.

CoQA, LlamaCoQA, GemmaCoQA, QwenTriviaQA with the passage, LlamaTriviaQA with the passage, GemmaTriviaQA with the passage, Qwen
LoRA Ensemble over meanings (best of its three scores)0.7500.7220.7230.6840.8160.770
CLLM (credal bound)0.8670.8730.7930.8440.8820.791

The credal bound detects corrupted context better than every score of this ensemble in all six settings (Table I), by up to 0.16; scored on its own MAP answer rather than the CSC representative, its ECE changes by at most 0.031 and its NLL by at most 0.051. It is described in §D.3 (L1191) and evaluated in Tab. 9 (§E.2, L1404) and Tab. 10 (§E.3, L1439). Against the decision (kKu7, W2), CSC with no threshold is as accurate as the best thresholded ensemble on Gemma and within 1.5 and 3.7 points on Llama and Qwen, and it commits on 3.6-7.6% of corrupted prompts, against 2.8-34.0% for the LoRA Ensembles.

Q3: We agree with your count and have corrected the limitation; the B partial sequences of a step go through each adapter together, in one batched pass.

A step runs at most B×MB \times M forward passes, fewer after pruning, so at most B×M×L=1,280B \times M \times L = 1{,}280 per prompt. The B partial sequences share the prompt and have equal length, so each adapter runs them in one batched pass with a key-value cache (§D.1, L1139-1142): M passes per step. §E.9 (L1935-1945) reports 68-89 decoding steps per prompt on CoQA and closed-book TriviaQA, at 7-12 s per prompt, plus 5-12 s for the equivalence judgements. "M times a single model" holds for CTC, which needs one forward pass per adapter (§3.4, L259), not for CSC, which adds the decoding and the equivalence judgements (L285). The limitation now reads: “CTC costs one forward pass per adapter, and credal decoding at most B forward passes per adapter per step (§E.9)” (L486-487).

Q4: Yes: the answers credal decoding finds are more accurate than greedy decoding, both of the base model and of the same five adapters, in every open-ended setting.

Credal decoding follows every partial sequence that no completed sequence dominates, up to the width B (§3.3, L227-238), and produces the completion set on which CSC decides; the paper is framed around that decision (see Suggestions). As the question is cut off, we answer whether credal decoding finds better answers: the CSC representative is more accurate than the Standard LLM's greedy answer in all nine open-ended settings (CoQA, TriviaQA with the passage and closed-book TriviaQA, three backbones), by 3-21 points, e.g. 83 vs 62.0% on TriviaQA with the passage on Gemma. We also decode greedily from the same five adapters, mean-pooled at every token (Table J).

Table J: Greedy decoding against credal decoding. Accuracy of the answer on all clean test prompts (%), 250 per cell. CSC representative: Acc (rep.) of Tab. 3. LoRA Ensemble greedy: the five adapters mean-pooled at every token, decoded greedily on the same prompts.

SettingStandard LLM, greedyLoRA Ensemble, greedyCSC representative (Tab. 3)
CoQA, Llama69.268.875
CoQA, Gemma71.272.478
CoQA, Qwen70.472.874
TriviaQA with the passage, Llama76.880.886
TriviaQA with the passage, Gemma62.070.483
TriviaQA with the passage, Qwen67.274.076
Closed-book TriviaQA, Llama72.872.877
Closed-book TriviaQA, Gemma67.268.876
Closed-book TriviaQA, Qwen52.851.656

Decoded greedily from the same five adapters, the answer is 1-13 points less accurate than the CSC representative in all nine settings. We are happy to answer the full question.

Q5 A truncation separates what the width B left out from what dominance ruled out, and the revised §3.3 now says so (L234-236).

It records (step 3, L234-235; worked example in §C.2, L1061-1063) that the width, not dominance, stopped partial sequences from being followed. Their mass is in umu_m either way (L247-252), so the bounds stay valid, and B itself is ablated in §E.7.2 (L252).

Suggestions and minor points

S1, S2: We agree: the thread is the decision of when to commit and what to return otherwise, and the paper is built as one interval-dominance rule at two depths, with credal decoding in its own section.

The goal is the decision: when the model should commit to an answer, and what it should return when it should not (Eq. (3), L205-207). Credal decoding produces the candidates, which are also more accurate than the base model's greedy answer (Q4); the lower bound of the committed answer is its confidence, tested in Tabs. 1 and 2; and the set is what the rule returns when it does not commit, the answers no plausible predictor rules out (L207). This is already the structure of the paper: the abstract derives one commitment rule applied at two depths (L16-19); the contributions define CTC and CSC as that rule on answer tokens and on meanings (L80-92); Fig. 2 applies one interval-dominance test, Eq. (3), in both spaces (L176); §3.4 states that CTC and CSC are the same rule on two candidate sets (L286); and credal decoding is a contribution of its own (L84-88) with its own section (§3.3, L221-254). Using the extra page, the decision results (Tab. 12 of the submission) are now Tab. 3 in the main text (L432), which keeps the paper centred on the decision.

Minor: We have fixed all three: the Fig. 3 caption now reads “The rule chooses the same operating point as a tuned threshold, and withdraws when the evidence is corrupted” (L386); the spacing around Tabs. 1 and 2 on page 7 is restored; and the text of §E.8 is no longer blue.

Thank you again for a review whose questions have made the paper clearer and its comparisons stronger. We hope these answers address your points, and that you will take them into account in your assessment. We would be glad to clarify anything further.

Reviewer_xtCuOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper proposes Credal Large Language Models (CLLMs), a credal-set framework built on LoRA ensembles. Using an interval-dominance rule, it commits to an answer only when one candidate’s lower probability exceeds all others’ upper bounds, and otherwise returns a set of plausible answers. It introduces credal decoding for open-ended generation, and applies the commitment rule at both token and semantic levels. Experiments across 3 backbones and 5 datasets show improved accuracy over single-model baselines, competitive performance vs mean-pooled ensembles, and reliable abstention under corrupted context without tuned thresholds. Key limitations include the small ensemble size, potential circularity in semantic judgment, and overly conservative handling of unexplored mass.Presentation

Critical strengths
  1. The paper brings interval dominance from imprecise probability theory to LLMs as a single, parameter-free mechanism to decide whether the model commits to an answer or returns a set of candidates. It naturally reduces to standard argmax prediction when all adapters agree, which is clean and easy to follow.

  2. The token-level (CTC) and semantic-level (CSC) commitment design well addresses the common case where adapters disagree on surface form but agree on meaning. Credal decoding with explicit unexplored mass makes the generation procedure theoretically sound, without needing sampling temperature or sample count.

  3. Experiments cover three backbones, five datasets, and multiple metrics, including selective accuracy, calibration, hallucination/corruption detection, and set-valued prediction. Detailed ablation studies and corrupted-context tests strongly support the conclusions.

Critical weaknesses
  1. All experiments use only M=5 adapters. The min/max bounds are sensitive to outlier adapters and may understate the true epistemic variation. The behavior of bounds and commit rates with larger M is not fully explored.

  2. The same ensemble that generates answers also judges semantic equivalence. This may bias the grouping toward the model’s own patterns. There is no external validation of the semantic clustering quality.

  3. Adding the full unexplored probability mass to the upper bound of every meaning is theoretically safe, but it is a very conservative choice. It may unnecessarily suppress commitment rates, especially in long-form or high-branching generation settings. The paper does not explore alternative ways to allocate this unmeasured probability.

Questions for authors
  1. You use only 5 LoRA adapters in all experiments. How will the probability bounds and commit rate change if you use more adapters? Have you run tests with larger M values?

  2. The same adapters both generate answers and judge if two answers mean the same thing. Could this make the semantic grouping biased? Will you add an external check for semantic equivalence in future work?

  3. You add all unexplored mass to the upper bound of every meaning, which is very conservative. Could this make the model commit less often than it reasonably should? Have you tried any softer ways to handle this unexplored probability?

2 comments

Dear Reviewer,

Thank you for your positive assessment and for your precise questions on the number of adapters, the equivalence judgement and the unexplored mass. We ran each of the requested checks, and answer them below.

W1, Q1: Thank you for suggesting larger ensembles; we have run them up to M=8, and the decision is stable: on Llama's CoQA development split, CSC commitment changes by at most 2 points from M=4 to M=8.

M=5 is compute-bounded (§4.1, L299), and §E.7.1 already varies M from 2 to 5 on all three backbones (L1709-1713, Tab. 17, Fig. 10). We have extended it to M=8 on Llama (Table E).

Table E: Number of adapters, M = 2 to 8. CoQA development split, 500 prompts, Llama-3.1-8B; adapters 6 to 8 vary target modules, rank, learning rate, resample and seed like the first five.

M2345 (paper)678
CSC commit rate (%)43.441.640.640.840.639.838.8
Accuracy when committed (%)89.991.393.192.292.192.593.3
Gold-in-set (%)83.887.689.889.289.889.489.8
Interval width0.540.550.580.580.570.590.62

The intervals widen with M, as stated at L213-215 (proof in §A.3), from 0.54 at M=2 to 0.62 at M=8, but the decision barely moves with them: accuracy when committed and gold-in-set saturate at M=4, as §E.7.1 found up to M=5 (L1710-1713; §4.2 (vi), L460-462). This is now Tab. 18 in §E.7.1 (L1725).

W2, Q2: Thank you for suggesting an external check; we have run it: with an external NLI partition in place of the adapters' judgement, accuracy when committed remains 92.6-100% across the nine clean test settings.

Two candidates are merged only when every adapter judges them equivalent (§3.4, L269-271), so a false merge requires all five adapters to make the same error. §E.7.3 already compares the grouping with the embedding clustering of semantic entropy and inspects 40 flagged prompts by hand (L1813-1821), and Tab. 22 contrasts the sure and possible partitions (L1824-1832). We have now replaced the adapters' judgement with the NLI clustering of semantic entropy (bidirectional entailment, DeBERTa-large-MNLI) and applied the rule unchanged to the same candidates, with the meaning masses and bounds recomputed. On all nine clean settings (CoQA, TriviaQA with the passage and closed-book TriviaQA, three backbones) accuracy when committed is 92.6-100% and gold-in-set changes by at most 0.8 points; on corrupted context commitment stays at or below 8.4%. This is now Tab. 23 in §E.7.3 (L1834).

W3, Q3: We agree that adding all of the unexplored mass is conservative: it is so by design, as the only choice that keeps a commitment valid. As you suggest, we have tested softer allocations, and they raise commitment where it is needed.

Adding all of it keeps a commitment valid whatever the completions credal decoding did not write out turn out to mean (Eq. (5), L277-283). We tested three softer allocations: half of each adapter's unexplored mass, a share in proportion to each meaning's mass, and none. A softer choice also lowers the bar against a meaning not yet written out: half and none apply the same allocation to it, and the proportional variant uses the largest share allocated to an observed meaning.

Table F: Allocating the unexplored mass. Clean test prompts; each cell: CSC commit rate / accuracy when committed (%).

Unexplored mass added to each meaning's upper probabilityCoQA, LlamaCoQA, GemmaTriviaQA with the passage, LlamaTriviaQA with the passage, GemmaTriviaQA with the passage, Qwen
all of it (paper)40.4 / 95.047.6 / 95.81.2 / 1001.2 / 1006.4 / 93.8
in proportion to the meaning's mass42.8 / 95.351.2 / 94.51.2 / 1001.2 / 1008.0 / 95.0
half of it52.8 / 93.253.6 / 94.86.8 / 10016.4 / 97.620.4 / 98.0
none66.0 / 90.360.8 / 91.423.6 / 10064.0 / 95.048.8 / 98.4

Adding half raises commitment by 5.6-15.2 points (Table F) with accuracy when committed at 93.2-100%, most on TriviaQA with the passage, where answers branch most (Gemma: 1.2% to 16.4% at 97.6%); a deployment that wants more coverage can use it. Dropping the mass shows why it is carried: commitment on corrupted CoQA rises from 3.6% to 14.4% (Llama) and 19.6% (Gemma), and the paper already reports that accuracy when committed on clean CoQA falls (§4.2 (vi), L463-466; §E.7.2). This is now Tab. 21 in §E.7.2 (L1780).

We thank you again for the positive assessment and the precise questions. The added experiments, on larger ensembles, an external equivalence judge and softer allocations of the unexplored mass, are all in the revised version; we hope they strengthen your confidence in the paper, and would be glad to clarify anything further.

Dear Reviewer,

As the discussion period comes to a close soon, we were wondering if you were happy with our detailed responses, corrections and experiments. Kindly let us know if you require any further clarification. We would highly appreciate your feedback.

Many thanks, Authors

Reviewer_kKu7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Credal Large Language Models (CLLMs) to improve uncertainty estimation and selective answering in LLMs. The method constructs a credal set from five independently fine-tuned LoRA adapters, using the minimum and maximum predicted probabilities to represent uncertainty. Based on interval dominance, the model commits to an answer only when it dominates all alternatives; otherwise, it returns a set of plausible answers. The authors extend this approach to open-ended generation through credal decoding and semantic clustering. Experiments across three LLM backbones and several QA benchmarks demonstrate improvements in uncertainty estimation, selective accuracy, and abstention under corrupted inputs compared with several baselines.

Critical strengths
  1. Important problem. Reliable confidence estimation and selective answering are important challenges for LLMs, particularly in factual question answering. Exploring richer uncertainty representations beyond a single predictive distribution is a worthwhile direction.

  2. Intuitive and interpretable approach. The use of probability intervals and interval dominance provides a principled and interpretable way to determine whether a model should commit to an answer or retain multiple alternatives.

  3. Reasonably comprehensive experiments. The paper evaluates across multiple QA benchmarks, compares against several relevant uncertainty-estimation baselines, and includes analyses of calibration, corrupted-input detection, commitment behavior, and methodological ablations.

Critical weaknesses
  1. Main evaluations do not directly assess the proposed commitment rule. Although Tables 1 and 2 compare against the same LoRA ensemble, they primarily evaluate scalar uncertainty scores derived from the credal set (e.g., the maximum lower probability and interval midpoint), rather than interval-dominance decisions. These results demonstrate the usefulness of credal-derived confidence scores but provide limited evidence for the proposed commitment mechanism itself.

  2. Insufficient comparison of actual commitment performance. While the paper reports commitment rates and accuracy when committed, these results are scattered across the appendix, with limited direct comparisons to baselines, particularly for open-ended QA. CSC also commits on only 1–6% of questions in some settings. Moreover, being threshold-free does not necessarily imply a better decision policy. A systematic comparison of accuracy and coverage against development-set-calibrated ensemble or self-consistency baselines, ideally at matched computational cost, would better establish the practical benefit of interval dominance.

  3. Insufficient justification and contextualization of the corruption setup. The paper does not clearly discuss how its corrupted-input evaluation relates to or differs from existing knowledge-conflict settings, such as ClashEval [1] and Situated Faithfulness [2], which also include confidence while the corrupted input conflicts with model parametric knowledge. Also, the closed-book corruption setup is unclear, replacing a question while retaining its original answer side may produce another perfectly valid, answerable question, for which increased uncertainty is not necessarily expected.

  4. Limited justification of ensemble disagreement as epistemic uncertainty. The proposed credal set is constructed from five LoRA adapters sharing the same backbone and underlying training data, with variation induced by bootstrap sampling and fine-tuning configurations. Such disagreement primarily captures sensitivity to training choices and may not reflect broader epistemic uncertainty, particularly when models share systematic errors or missing knowledge. The paper does not isolate the sources of ensemble diversity or establish whether the resulting uncertainty bounds meaningfully capture factual ignorance. Controlled diversity and data-size ablations, together with evaluations of shared model misconceptions, would strengthen this interpretation.

[1] Wu et al. (2024). ClashEval: Quantifying the Tug-of-War Between an LLM’s Internal Prior and External Evidence. NeurIPS 2024.

[2] Huang et al. (2025). To Trust or Not to Trust? Enhancing Large Language Models’ Situated Faithfulness to External Contexts. ICLR 2025.

Questions for authors

N/A

4 comments

Dear Reviewer,

We thank you for your careful review and for the concrete suggestions on evaluating the decision itself and on the sources of ensemble diversity. We ran the analyses you asked for, and answer each point below:

W1: We respectfully disagree that the commitment rule is not assessed directly: Tabs. 1 and 2 evaluate the quantities the rule decides from, and the decision itself is evaluated in Tabs. 3, 12, 14 and 27 and Figs. 3, 4 and 7.

The rule commits to a∗a^* when its lower probability is at least the upper probability of every alternative (Eq. (3), L205), and reports that lower probability as its confidence (L206). Tab. 1 tests whether it separates right from wrong answers and intact from corrupted inputs; Tab. 2 tests how accurate answers ranked by it are, and whether the interval midpoint is calibrated. The baselines have no decision of their own, only a score that needs a threshold.

The decision is evaluated by commit rate, accuracy when committed, gold-in-set and set size, on clean and corrupted context (metrics in §4.1 from L326; results in §4.2 (ii)-(v)). On CoQA and closed-book TriviaQA, CSC is right on 91-96% of the questions it commits to, against 56-78% for the same answers over all questions; corrupting the context lowers its commitment on CoQA from 31-48% to 4-8%; and CTC commits on 73-91% of four-option questions at 89-98% accuracy, with sets of 1.1-1.5 options that contain the gold option on 89-98% (Tab. 12, the four-option results); all with no tuned threshold.

W2:

2. Insufficient comparison of actual commitment performance. While the paper reports commitment rates and accuracy when committed, these results are scattered across the appendix, with limited direct comparisons to baselines, particularly for open-ended QA.

We agree that the decision results deserved a central place, and have moved them into the main text.

Using the extra page, Tab. 12 of the submission, which shows commitment selecting the questions the model answers correctly and withdrawing under corrupted context, is now Tab. 3 in the main text (L432), and the harmful-prompts table has moved to §E.10 (Tab. 28). On open-ended QA, Figs. 3 and 7 already compare the decision with the baselines at the rule's own coverage (CoQA and TriviaQA, all three backbones), and the comparison below adds development-calibrated baselines (Table A).

Moreover, being threshold-free does not necessarily imply a better decision policy. A systematic comparison of accuracy and coverage against development-set-calibrated ensemble or self-consistency baselines, ideally at matched computational cost, would better establish the practical benefit of interval dominance.

We agree that being threshold-free is not by itself a better policy, and thank you for the suggested comparison, which we have added: CSC chooses a competitive operating point itself, without fitting an abstention threshold.

Each baseline is thresholded to commit on CSC's fraction of clean prompts (the LoRA Ensembles on the CoQA development split; the single-model baselines, which have no development run, by two-fold cross-fitting on the test prompts), and the same threshold is applied to corrupted prompts. The LoRA Ensemble over meanings comes from the same credal decoding run as CSC, so its cost is identical, and the single-model baselines use their standard budgets (K=16); for predictive entropy and mutual information the ensemble's unexplored mass counts as one more outcome.

Table A: Commitment against thresholded baselines. CoQA, 250 clean and 250 corrupted prompts. Each cell: accuracy when committed on clean prompts / coverage on clean prompts / commit rate on corrupted prompts (%). Each threshold targets CSC's coverage; self-consistency and semantic entropy are 0 whenever all 16 samples agree, so their threshold falls on that tie and their coverage cannot be set to CSC's.

Method (threshold)Llama-3.1-8BGemma-2-9BQwen2.5-7B
Standard LLM, predictive entropy (cross-fitted)93.1 / 40.8 / 16.487.7 / 48.8 / 8.890.8 / 30.4 / 16.2
Self-consistency, K=16 (cross-fitted)80.0 / 42.0 / 25.484.1 / 52.8 / 19.281.3 / 42.8 / 14.8
Semantic entropy, NLI, K=16 (cross-fitted)84.5 / 41.2 / 18.684.1 / 52.8 / 19.281.3 / 42.8 / 14.8
LoRA Ensemble, first-token predictive entropy, as in Tabs. 1 and 2 (development split)96.5 / 45.2 / 4.093.7 / 50.8 / 5.294.6 / 29.6 / 5.6
LoRA Ensemble over meanings, predictive entropy (development split)85.7 / 44.8 / 31.690.6 / 46.8 / 34.091.2 / 32.0 / 10.0
LoRA Ensemble over meanings, mutual information (development split)87.6 / 42.0 / 12.092.7 / 49.6 / 15.688.5 / 31.2 / 7.6
LoRA Ensemble over meanings, 1 − probability of the MAP meaning (development split)96.1 / 40.8 / 2.895.8 / 48.0 / 3.693.4 / 30.4 / 8.8
CSC (no threshold)95.0 / 40.4 / 3.695.8 / 47.6 / 3.690.9 / 30.8 / 7.6

With no threshold of its own, CSC is as accurate as the best threshold on Gemma (Table A), within 1.5 points on Llama and 3.7 on Qwen, and 9.6-15.0 points above self-consistency and semantic entropy. On corrupted context it commits on 3.6-7.6% of prompts, as rarely as the strongest ensemble scores (2.8-8.8%) and well below the single-model baselines (8.8-25.4%) and the LoRA Ensemble over meanings with predictive entropy (up to 34.0%). Each baseline needs a threshold set to the coverage CSC chose; CSC chooses that operating point itself, and it carries over to corrupted context. This is now Tab. 10 in §E.3 (L1439-1446); the baseline is described in §D.3 (L1191).

CSC also commits on only 1–6% of questions in some settings.

We disagree that this is a failure of the rule: on TriviaQA with the passage it declines because much of the probability mass is unexplored, and when it commits it is right on 94-100% of questions.

This happens only on TriviaQA with the passage, whose long-tail answers have many surface forms: the unexplored mass is 0.78-0.80 (Tab. 3). It is added to the upper probability of every meaning (Eq. (5), L277-283), which usually prevents any meaning from dominating, so the rule rarely commits to an answer whose support it has not seen (§4.3, L483-485). Its sets contain the gold meaning on 86-91% of these questions (Tab. 3). Elsewhere CSC commits on 30.8-47.6% of CoQA and 24.4-45.6% of closed-book TriviaQA questions, and CTC on 73-91% of four-option questions. Adding half of the unexplored mass instead raises commitment on TriviaQA with the passage to 6.8-20.4% at 97.6-100% accuracy (xtCu, W3).

W3:

3. Insufficient justification and contextualization of the corruption setup. The paper does not clearly discuss how its corrupted-input evaluation relates to or differs from existing knowledge-conflict settings, such as ClashEval [1] and Situated Faithfulness [2], which also include confidence while the corrupted input conflicts with model parametric knowledge.

We respectfully disagree that the setup is unjustified: it asks a different question from ClashEval and Situated Faithfulness.

Those benchmarks ask which source a model should trust when the context conflicts with its parametric knowledge, and score whether it picks the right one; the model is still expected to answer. We ask whether the model should answer at all. Where the answer rests on supplied evidence, the evidence is replaced and the question kept, so the question stays answerable in principle but the evidence no longer supports an answer (§D.4, L1217-1220; Tab. 1 caption, L336-337), and the rule is scored on whether it withdraws (Tab. 3, Fig. 3).

Also, the closed-book corruption setup is unclear, replacing a question while retaining its original answer side may produce another perfectly valid, answerable question, for which increased uncertainty is not necessarily expected.

This does not weaken the evaluation: questions are replaced only in two detection columns of Tab. 1, and every result on the decision keeps the question unchanged.

Questions are replaced only where there is no evidence to replace, in the corrupted columns of Tab. 1 for closed-book TriviaQA and ARC-Challenge (caption, L336-337). Every other corrupted-context result replaces the evidence and keeps the question: Tab. 3 and Fig. 3 (commitment), Tab. 14 (token space), Fig. 4 and Tab. 12 (four-option), and Fig. 5 and Tab. 8 (detection) (§D.4, L1217-1220). These two columns measure detection of question replacement, not whether the replacement question is unanswerable; there CLLM reaches 0.701 and 0.783, against 0.472-0.556 for the single-model scores.

W4: We thank you for suggesting these three analyses, which we have run. They show that the adapters' disagreement tracks whether the answer is right.

The credal set is the convex hull of the predictive distributions of plausible fine-tunings of the backbone on the same data (§3.1, L186-187), which vary target modules, rank, learning rate, bootstrap resample and seed (§4.1, L301-302; §D.1).

(a) Sources of diversity. Ensembles whose adapters differ only in seed, or in seed and bootstrap resample (target modules q and v, rank 8, learning rate 10⁻⁴), against the paper's ensemble.

Table B: Sources of ensemble diversity. CoQA development split, 500 prompts, M=5.

BackboneAdapters differ inInterval widthGold-in-set (%)CSC commit rate (%)Accuracy when committed (%)
Llama-3.1-8Bseed only0.4178.660.486.4
Llama-3.1-8Bseed and bootstrap resample0.4278.660.086.0
Llama-3.1-8Btarget modules, rank, learning rate, resample, seed (paper)0.5889.240.892.2
Qwen2.5-7Bseed only0.2074.280.077.8
Qwen2.5-7Bseed and bootstrap resample0.2074.481.077.0
Qwen2.5-7Btarget modules, rank, learning rate, resample, seed (paper)0.6588.829.093.8

These adapters are more alike (Table B): narrower intervals (0.41-0.42 and 0.20, against 0.58 and 0.65), more frequent commitment and less accurate committed answers. With the paper's adapters the set contains the gold meaning 10.6-14.6 points more often, and accuracy when committed is 5.8-16.8 points higher.

(b) Training-set size. The paper's ensembles trained on 10% and 30% of the training split.

Table C: Training-set size. CoQA development split, 500 prompts, M=5.

Training dataLlama-3.1-8B: gold-in-set / accuracy when committed (%)Qwen2.5-7B: gold-in-set / accuracy when committed (%)
10%85.0 / 90.984.0 / 87.2
30%87.0 / 92.985.6 / 89.9
100% (paper)89.2 / 92.288.8 / 93.8

With more data the set contains the gold meaning more often on both backbones (Table C), and accuracy when committed is higher than at 10%.

(c) Shared misconceptions.

Table D: Shared misconceptions. Clean test prompts; a question is "all wrong" when each adapter's own most probable answer is wrong.

CoQA, LlamaCoQA, GemmaCoQA, QwenClosed-book TriviaQA, LlamaClosed-book TriviaQA, GemmaClosed-book TriviaQA, Qwen
All five adapters wrong: CSC commit rate (%)13.212.515.98.512.23.3
All five adapters right: CSC commit rate (%)56.770.145.652.673.460.0
Share of all commitments on all-wrong questions (%)6.94.29.15.05.34.9

When all five adapters are wrong their answers usually differ (Table D), and CSC commits on 3.3-15.9% of those questions, against 45.6-73.4% when all five are right; fewer than one commitment in ten falls on such a question. All three are in the new §E.7.4 (L1863-1872, Tabs. 24-26), and §D.1, which describes the ensemble, now points to the diversity comparison (L1136-1138).

We thank you again. We have run all the analyses you requested, and moved the decision results into the main text; we hope these address your concerns, and that you will consider them in the final assessment. We would be glad to clarify anything further.

Dear Reviewer,

As the discussion period comes to a close soon, we were wondering if you were happy with our detailed responses, corrections and experiments. Kindly let us know if you require any further clarification. We would highly appreciate your feedback.

Many thanks, Authors

Comments on the paper

1 comment

Dear Reviewers and Area Chair,

We have uploaded a revised version of the paper with minimal changes as per the suggestions of the reviewers. All edits are marked in blue.

  • No new experiments beyond the ablations requested by reviewers (all in the appendix, only added for better reading and more clarity): The new tables report the analyses asked for in the reviews: commitment against calibrated baselines (Tab. 10) and the sources of ensemble diversity, training-set size and shared errors (Tabs. 24–26), requested by Reviewer kKu7; ensembles of up to eight adapters (Tab. 18), an external equivalence judge (Tab. 23) and softer allocations of the unexplored mass (Tab. 21), requested by Reviewer xtCu; and length-normalised pruning (Tab. 20) and the LoRA Ensemble over meanings (Tab. 9), requested by Reviewer nDEU.

  • One existing table moved into the main text. As Reviewer kKu7 noted that the commitment results were scattered across the appendix, the Credal Semantic Commitment table (Tab. 12 of the submission) is now Tab. 3 in the main text, using the extra page. To make room, the harmful-prompts table (Tab. 3 of the submission) has moved to §E.10 (Tab. 28) for now; if space allows in the camera-ready version, we will bring the harmful-prompts table back to the main text. The contents of both tables are unchanged; the tables are renumbered accordingly, and our replies use the revised numbering.

  • Other changes to the main text are limited to those stated in our replies: two clarifications in §3.3, the Fig. 3 caption, the limitations, and the spacing around Tabs. 1 and 2.

We thank all three reviewers for their careful reviews, and hope that the revised version and our replies address your questions. We would be glad to clarify anything further during the discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.