acceptodds
Desk rejected at ICLR 2027

XU-RS: Explaining Credal Width in Random-Set Language Models

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Uncertainty estimates tell us how unsure a model is, but not why. Without knowing which parts of an input influences a model's uncertainty, we cannot tell whether that uncertainty score depends on input features that are relevant for the task. We study this problem in random-set classifiers built using pretrained language models. These classifiers assign probability to individual answers and to groups of answers, producing lower and upper probabilities for each answer; The difference between these probabilities, called credal width, is used to represent epistemic uncertainty about an answer arising from limited training data. We propose XU-RS, a framework that attributes an answer's credal width to the input tokens (words or word pieces) supplied to a language model. XU-RS uses Expected Gradients (a standard feature attribution method) to estimate how input tokens contribute to credal width. The proposed framework is evaluated on a MedQA dataset using SmolLM3-3B and Llama-2-7B models, demonstrating that setting the embedding of a token ranked highly by XU-RS to zero (zero-masking) causes larger changes in credal width than zero-masking randomly selected tokens. In addition, we show that normalisation can cause other answer groups to influence an answer's width, reveal how token attribution can mask numerical errors, and provide diagnostic checks to verify whether a token ranked highly by XU-RS meaningfully explains model uncertainty.

Primary area:
interpretability and explainable AI
Keywords:
uncertainty attribution, uncertainty quantification, interpretability, random sets, credal sets, feature attribution, belief functions, large language models, epistemic uncertainty

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_TSxMOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper applies a random-set head to a pre-trained LLM and uses gradient attribution to quantify which tokens in the input drive the decision. They evaluate on one dataset and show that zero-masking the tokens attributed by method correlates better with uncertainty (credal width) than zero-masking random tokens in the input.

Critical strengths

I can not identify any clear strength in this paper.

Critical weaknesses
  • The paper's writing severely lacks focus. Methodology and results are just repeated iterations of numbers without any motivation, interpretation, or intuition whatsoever. This makes getting the gist of the experiments and the significance of the results very difficult. Furthermore, the paper weights numerical verification of insensitivity to sampling in their attribution method as strongly as the main result (advantage over zero masking).
  • The paper's novelty and impact is heavily limited: Gradient-based attribution already exists and the paper mostly just applies this to an LLM-based classifier. Also random-set classifiers are not novel. The paper only considers one dataset and the main experiment has one baseline only (zero masking). Overall this paper feels more like a case study than novel scientific results and ideas. I will acknowledge however that I am not the most familiar with the related work in this field.
  • Many architectural choices seem somewhat arbitrary and are not well motivated in the paper (see questions).
Questions for authors
  • What is the motivation for using gradient attribution jointly with random-set prediction? Are there any expected advantages from combining the two in theoretical or practical terms?
  • L 200f.: How does the final correction that ensures probabilities are well-behaved look like formally?
  • Is token-by-token comparison between original and reference embeddings a good idea in general? Doesn't that disregard the semantics?
  • What are the prepared references? How is a reference for an input chosen?
  • Mass correction and credal width: How are these calculations justified / motivated? Do they come from existing work (then they lack citations)?
  • Attribution score: Why are you sampling 20 eligible positions instead of just averaging over all available eligible positions?
  • L. 340: My understanding was that the classification head predicts 2^4 scores using sigmoid activations for each set and then corrects those to get final probabilities. Here you now talk about applying a softmax to 15 of them? What is the rationale? What are you trying to achieve?
  • Do you have any examples of questions and their highest attributed tokens for some intuitive understanding of what your method actually produces as explanations beyond just high interval chance?
  • L. 301: You state that you report error detection AUC-ROC. Where are these numbers to be found? What baselines are you comparing against?
1 comment

Two statements in the summary differ from the paper. The result is not a correlation with uncertainty: it is the absolute change in the selected answer's width when a token's embedding is set to zero. Zero-masking is the operation applied in both arms; the comparison is between the highest-ranked token and random tokens.

W1. We agree the paper gives the numerical checks as much space as the MedQA result without saying which claim each supports, and will restructure.

W2. See point 2 of the general response.

Q1. See point 1 of the general response.

Q2. Eq. 5 and Appendix A.2: with intermediate values m̃_S, set q_S = max(0, m̃_S), s = Σ q_S over non-empty proper subsets and r = max(1 − s, 0); then m(S) = q_S/(s + r) and m(Θ) = r/(s + r).

Q3. It is a real limitation, noted in Section 2 (lines 147–150): interpolating position by position between unrelated prompts passes through vectors that correspond to no text. It is inherent in Expected Gradients with reference prompts, and Section 6.4 measures one consequence: fitting a reference to the input's positions changed the explained width difference by 0.20–0.54.

Q4. Section 3.3 and Appendix B.2. References are not chosen per input: the first 256 MedQA training prompts for SmolLM3, and three disjoint sets of 1,024 training prompts in a fixed hashed order for Llama. Each is truncated or padded to the input's length and evaluated with the input's mask and positions.

Q5. The conversion and correction follow RS-NN and RS-LLM, cited where they are introduced (lines 190–198). The closed form for width after correction (Eq. 6) is ours. We will repeat the citation in Section 4.

Q6. Twenty draws was a compute choice. We have now replaced every eligible token, which gives the exact mean over tokens; the advantages are unchanged (0.038 and 0.045). See point 4.

Q7. These are two different models and the paper should separate them more clearly. The MedQA classifiers have 14 sigmoid outputs, converted and corrected as in RS-LLM. The model used for the synthetic checks applies a softmax over the 15 non-empty answer sets, which gives valid masses directly, so that attribution numerics could be tested without the correction step.

Q8. See point 3 of the general response.

Q9. Section 6.1, lines 401–403: width gives AUROC 0.634 (SmolLM3) and 0.598 (Llama) for detecting errors on the full test set; the baseline is low pignistic probability of the selected answer, 0.680 and 0.671. We will move these into a table.

Reviewer_yCPuOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The authors for this paper study why a random-set language model is uncertain about an answer, not just how uncertain it is. The classifiers put probability on groups of answers, which gives each answer a lower and upper probability and the gap is the uncertainty. They use Expected Gradients, a standard attribution method, to score how much each input token contributes to this width. On MedQA questions with SmolLM3-3B and Llama-2-7B, blanking out the top-scored token changes width more than blanking out random tokens.

Critical strengths
  • Asking why a model is uncertain, and not only how much, is a meaningful question, especially for medical use.
  • The findings are a useful for using attribution on text, showing that stable token rankings can hide wrong attribution values, and preparing the reference prompt can change what is being explained.
  • The main result is reported with bootstrap confidence intervals.
Critical weaknesses
  • XU-RS takes an existing method and applies it to a new output. Expected Gradients is used as is, and earlier work has already used attribution to explain uncertainty (e.g., Perez et al., 2022, Wang et al., 2023, Watson et al., 2023, Iversen et al., 2025). Checking completeness and the choice of reference is also common practice when using Integrated Gradients. What is new is mainly the decision to explain credal width.

  • Width is worse at detecting errors (AUROC) than the plain probability of the chosen answer, and the classifiers are not strong on four-option MedQA. Test predictions have negative values that should be corrected, and rescaling happens on 96–99.9% of questions as shown in Table 8, so this means the width depends on the correction step and not only on what the model learned. Hence, it is unclear why this width is the right thing to explain.

  • Top-ranked tokens are compared only with randomly chosen tokens, and as the authors acknowledge, these are not matched by position or by section of the prompt. Also, blanking a token removes its entire embedding, so tokens with larger embeddings might shift width more no matter how they are ranked. So, ranking tokens by embedding size would be a check to test.

  • The main experiment covers only the first 100 test questions for each model, uses one training seed, and masks only the single highest-ranked token. The average improvement is around 0.04, but without knowing the typical width or the typical effect of masking a random token, it is difficult to tell whether this effect is large or small. The numerical and reference results are based on just four synthetic examples, so overall the amount of evidence is narrow.

  • The paper does not show that the highlighted tokens carry meaningful information. The synthetic dataset contains a shortcut, the models trained on the revised dataset did not pass the checks, and no experts or other people review the MedQA explanations. The two models are also run with different settings which makes them hard to compare, and the code is not available.

Questions for authors
  • When blanking the top token changes the model's predicted answer, you measure the original answer's width. How often does this occur?
  • What tokens rank highest on MedQA, for example clinical findings, numbers, or answer option text?
1 comment

We thank the reviewer for a careful reading.

W1. Agreed; see point 2 of our general response.

W2. Agreed that width here depends on the correction step; this is what Section 4 and Table 8 document. We did not measure how much of the attribution flows through the shared total.

W3. See the table in point 4 of the general response. The advantage survives matching on prompt part and on position under zero-masking in both models and under padding replacement for SmolLM3; it does not for Llama under padding. Within a single option or the stem it is smaller but above zero under zero-masking (0.017 [0.005, 0.030] and 0.015 [0.004, 0.027]). Embedding size does not explain it: zeroing the token with the largest embedding changes width by 0.004 [−0.005, 0.014] and 0.009 [0.000, 0.019] more than an average token, and its rank correlation with the masking effect is −0.01 and 0.07.

W4. Mean width on the panels is 0.18–0.20, with 27 and 30 questions below 0.01. The top token changes width by 0.063 and 0.076 on average (35–39% of mean width); a random token by 0.025 and 0.031 (14–16%). Budgets of one to ten tokens were run (point 4). The attribution study used one checkpoint per model (seed 7); the other seeds were used only for Table 8.

W5. Agreed; point 3 of our general repsonse gives the description of the highlighted tokens that the paper lacked. The two models use different attribution budgets because a single reference collection failed our ranking checks for Llama (Appendix B.6); we agree this limits comparison between them.

Q1. We measured this in the new run. Zero-masking the top-ranked token changes the predicted answer in 29% (SmolLM3) and 28% (Llama) of questions, against 4.8% and 8.1% for an average eligible token. Under padding the rates are 30% against 5.4%, and 64% against 30%. The top token's masking also moves the answer probability more than it moves width (0.088 and 0.070 above an average token, against 0.038 and 0.045). The highlighted tokens are therefore tokens the answer depends on, which fits the finding in point 4 of our general response that the ranking is not specific to width.

Q2. See point 3 of the general response.

Reviewer_jcQ1OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

My understanding of this paper is:

  1. they look at a classification problem (medqa) with 4 options
  2. they then recast this a problem where they predict which of 2**4-2 non-empty subsets of the possible answers occur
  3. this subset prediction problem is solved via some existing approach that I didn't follow too closely
  4. then, they define the "credal width" of a given answer ("c") as the sum of all probabilities assigned to sets containing that answer
  5. they then run GradientShap on this number to attribute "uncertainty" to tokens

The mathematization of this paper---the definition of the credal width as a measure of uncertainity and the use of the attribution method---both feel arbitrary to me. I see no reason, either theoretical or empirical, to prefer this over the standard softmax probability. I don't see any empirical results that suggest the approach succeeds either as a method of uncertainity quantification or as a method for attribution.

Critical strengths

The paper is honest about its negative empirical findings.

Critical weaknesses

Fundamentally, the premise of "uncertainity attribution" via the credal width + expected gradient method seems basically arbitrary to me. I can't imagine a situation in which an actual decision would be based on such a method. The paper does not offer either insight into why we might believe this is a fruitful direction nor empirical evidence that is reasonable.

Beyond this fundamental problem, an LLM offers the following additional criticisms, all of which seem serious:

  1. Their "theoretical contribution" (Section 4, pp. 5–6) is just the quotient rule of calculus: • Because the 14 sigmoid outputs aren't constrained to be valid set probabilities, the unnormalized positive masses q_S sum to s > 1 on 96%–99.9% of test examples (Section 6.2, p. 8). • Therefore, W_c = u_c/s, where u_c = ∑_{S∋c,|S|>1} q_S. Section 4 solemnly "derives" that differentiating W_c = u_c/s gives ∇W_c = 1/s ∇u_c - u_c/s² ∇s (Eq. 7, p. 6), and calls it a finding that normalizing across sets makes W_c depend on sets not containing c.
  2. The uncertainty metric they are explaining (W_c) is worse than standard softmax probability: • On MedQA, ranking predictions by credal width W_c detects wrong answers with AUROC 0.634 (SmolLM3-3B) and 0.598 (Llama-2-7B), which is worse than simply using the predicted answer's probability (0.680 and 0.671, Section 6.1, p. 8).
  3. Their main empirical result only beats random token masking (Section 6.1, Table 1, p. 8): • On 100 MedQA questions, zeroing out the embedding of the #1 token ranked by XU-RS changes W_c by 0.038–0.045 more than zeroing out a uniformly random token. Almost any gradient norm will beat random token masking; they never compare against vanilla gradient norm ‖∇_{xᵢ}W_c‖₂ or attributing standard entropy/confidence.
  4. Their synthetic validation experiment completely collapsed (Section 5.4, p. 7 & Appendix D, pp. 18–20): • Because zero-masking doesn't show that the highlighted tokens are meaningful clinical evidence, they generated a synthetic medical dataset (V1) with known diagnostic clues. • Later, they discovered a bug in V1: whichever diagnostic test was mentioned first in the prompt was 100% predictive of the label (Appendix D.1, p. 19)! And even on the 1 example they analyzed (Table 9, p. 18), standard XU-RS ranked the true diagnostic clue at 44th place (40th percentile—worse than average!). • When they fixed the dataset bug (V2), their models failed to learn the task on unseen templates (0.43 total probability error, Table 10, p. 20), so they had to salvage the paper by using 4 examples from the broken V1 model purely to check whether Captum's stochastic GradientShap integral converged numerically (Section 6.3–6.4, pp. 8–9).
Questions for authors

None

1 comment

Summary. Two corrections. Width is the mass on sets containing the answer together with at least one other option (Eq. 2), not on all sets containing it. And the choice is not arbitrary; see point 1 of the general response.

"I see no reason to prefer this over the softmax probability." The paper does not claim width should be preferred, and reports that it detects errors less well. Error detection is also not what width is meant to measure: lower and upper probabilities describe how much probability the model leaves uncommitted between answers, which a single probability vector cannot express (Appendix A.4 gives two predictions with identical answer probabilities and widths of one and zero). Whether trained models use that capacity in a useful way is the open question, and we agree our results do not show that they do.

Section 4 is the quotient rule. Agreed that the derivative is elementary; we will present it as a remark. Its content is the consequence, stated in point 1: on almost every test question the explained quantity is u_c/s.

Only beats random masking. Agreed, and the new comparison in point 4 goes into it further. Expected Gradients of width beats embedding norm and position, but not a single gradient of width, and the gradient of the answer probability or of entropy selects tokens whose masking changes width as much or more. We compared against gradients of those targets, not against Expected Gradients of them.

Synthetic study. The description of the first dataset's shortcut and the revised dataset's failed checks is accurate, and both are reported in the paper. Three statements go further than the paper supports:

  • Table 9's rank 44 is not Expected Gradients. It is Integrated Gradients against training references on one example, rank 44 of 205 tokens. We report it as a failure to localise the clue.
  • The four synthetic cases were not used "purely to check whether Captum's stochastic integral converged". The reference-preparation result (Table 3) uses complete Integrated Gradients integrals and concerns what difference is explained, not convergence.
  • Sections 6.3–6.4 do not depend on the model having learned the task. They test whether attribution values account for a stated output difference, which holds for any differentiable model. Appendix D says this.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.