- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper applies a random-set head to a pre-trained LLM and uses gradient attribution to quantify which tokens in the input drive the decision. They evaluate on one dataset and show that zero-masking the tokens attributed by method correlates better with uncertainty (credal width) than zero-masking random tokens in the input.
Critical strengths
I can not identify any clear strength in this paper.
Critical weaknesses
- The paper's writing severely lacks focus. Methodology and results are just repeated iterations of numbers without any motivation, interpretation, or intuition whatsoever. This makes getting the gist of the experiments and the significance of the results very difficult. Furthermore, the paper weights numerical verification of insensitivity to sampling in their attribution method as strongly as the main result (advantage over zero masking).
- The paper's novelty and impact is heavily limited: Gradient-based attribution already exists and the paper mostly just applies this to an LLM-based classifier. Also random-set classifiers are not novel. The paper only considers one dataset and the main experiment has one baseline only (zero masking). Overall this paper feels more like a case study than novel scientific results and ideas. I will acknowledge however that I am not the most familiar with the related work in this field.
- Many architectural choices seem somewhat arbitrary and are not well motivated in the paper (see questions).
Questions for authors
- What is the motivation for using gradient attribution jointly with random-set prediction? Are there any expected advantages from combining the two in theoretical or practical terms?
- L 200f.: How does the final correction that ensures probabilities are well-behaved look like formally?
- Is token-by-token comparison between original and reference embeddings a good idea in general? Doesn't that disregard the semantics?
- What are the prepared references? How is a reference for an input chosen?
- Mass correction and credal width: How are these calculations justified / motivated? Do they come from existing work (then they lack citations)?
- Attribution score: Why are you sampling 20 eligible positions instead of just averaging over all available eligible positions?
- L. 340: My understanding was that the classification head predicts 2^4 scores using sigmoid activations for each set and then corrects those to get final probabilities. Here you now talk about applying a softmax to 15 of them? What is the rationale? What are you trying to achieve?
- Do you have any examples of questions and their highest attributed tokens for some intuitive understanding of what your method actually produces as explanations beyond just high interval chance?
- L. 301: You state that you report error detection AUC-ROC. Where are these numbers to be found? What baselines are you comparing against?
Two statements in the summary differ from the paper. The result is not a correlation with uncertainty: it is the absolute change in the selected answer's width when a token's embedding is set to zero. Zero-masking is the operation applied in both arms; the comparison is between the highest-ranked token and random tokens.
W1. We agree the paper gives the numerical checks as much space as the MedQA result without saying which claim each supports, and will restructure.
W2. See point 2 of the general response.
Q1. See point 1 of the general response.
Q2. Eq. 5 and Appendix A.2: with intermediate values m̃_S, set q_S = max(0, m̃_S), s = Σ q_S over non-empty proper subsets and r = max(1 − s, 0); then m(S) = q_S/(s + r) and m(Θ) = r/(s + r).
Q3. It is a real limitation, noted in Section 2 (lines 147–150): interpolating position by position between unrelated prompts passes through vectors that correspond to no text. It is inherent in Expected Gradients with reference prompts, and Section 6.4 measures one consequence: fitting a reference to the input's positions changed the explained width difference by 0.20–0.54.
Q4. Section 3.3 and Appendix B.2. References are not chosen per input: the first 256 MedQA training prompts for SmolLM3, and three disjoint sets of 1,024 training prompts in a fixed hashed order for Llama. Each is truncated or padded to the input's length and evaluated with the input's mask and positions.
Q5. The conversion and correction follow RS-NN and RS-LLM, cited where they are introduced (lines 190–198). The closed form for width after correction (Eq. 6) is ours. We will repeat the citation in Section 4.
Q6. Twenty draws was a compute choice. We have now replaced every eligible token, which gives the exact mean over tokens; the advantages are unchanged (0.038 and 0.045). See point 4.
Q7. These are two different models and the paper should separate them more clearly. The MedQA classifiers have 14 sigmoid outputs, converted and corrected as in RS-LLM. The model used for the synthetic checks applies a softmax over the 15 non-empty answer sets, which gives valid masses directly, so that attribution numerics could be tested without the correction step.
Q8. See point 3 of the general response.
Q9. Section 6.1, lines 401–403: width gives AUROC 0.634 (SmolLM3) and 0.598 (Llama) for detecting errors on the full test set; the baseline is low pignistic probability of the selected answer, 0.680 and 0.671. We will move these into a table.