- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies uncertainty quantification in diffusion LLMs (dLLMs). They create a dataset of questions, each of which falls into one of four different kinds of uncertainty: task underspecification, context underspecification, epistemic, and aleatoric. They find that dLLM diffusion trajectories are distinct across these different kinds of uncertainty, and that simple temporal statistics over these trajectories can predict which kind of uncertainty the model is confronted with, as well or better compared to a CNN-based probe.
Critical strengths
- I think this is an interesting problem domain, and to my knowledge this work is novel in its dataset and approach
- The approach is simple but effective, and I find the results of Figure 3 especially compelling
Critical weaknesses
- Writing and presentation could be improved. My reading is that this paper was partly written/edited with AI, and it shows in a number of places.
- I'm a little unsure about how well the dataset generalizes to real world tasks, or whether particular categories measure what they claim to. In particular:
- It seems odd that aleatoric uncertainty equates to questions with multiple valid answers ("Name three sports teams in Boston") instead of questions with one true answer that is impossible to know ("I flipped a coin - did it come up heads or tails?"). I could be convinced that the former is valid, but this feels a bit taken for granted.
- It seems like task underspecification may be a specific kind of context underspecification
- What about cases where an input prompt has multiple kinds of uncertainty, or where it's ambiguous from the prompt what kind of uncertainty there is?
- I would have liked to see this applied to real world tasks as well as the authors' constructed dataset. It would be interesting if this kind of analysis could apply to arbitrary prompts to identify what kind of uncertainty that prompt might be encoding.
- I also would have liked to see quantitative analysis of temporal dynamics, rather than the qualitative descriptions in section 5.1
Questions for authors
- What are the "interpretable temporal statistics" that you measured (lines 456-7) - is this just the plots shown in Fig 3?
- How do you compute the commit step ? (line 299)
- Could you provide a more formal definition of what posterior distribution the "posterior cube" represents?
- What does in figure 1 represent?
- Can your analysis of denoising dynamics help us identify cases with multiple kinds of uncertainty, or where the model is uncertain about what kind of uncertainty there is?
4 comments
Thank you again for your review. As the discussion period closes, we wanted to follow up on our rebuttal. We’d welcome any further feedback or remaining concerns, and hope you’ll consider our response in your final assessment.
W1: Fair enough. I will say, that as a native english speaker, in general I'd much rather read text written by a non-native speaker than text written/heavily edited by an LLM.
I do wish there was a good recipe for making writing not sound LLM-y, work like [1, 2] may be a helpful reference, but if there was good advice for this I would give it to 4/5 papers I was assigned to review this cycle. That said, the writing here is the least LLM-y of those 4 papers, but a good example of the AI accent is the figure 2 caption - e.g. "a detail only the user holds" is an unusual way to say "a fact known only by the user" (also, does the user necessarily know this? anyways...), and "chained aggregations collapse the cube axis by axis" is a complex phrase that probably needs some unpacking for the average reader. Similar comment for "supply questions that need a detail only the user holds" in the main text - some LLMs seem to really like phrases like "holds", "carries", "load-bearing", etc.
W2: Thank you for clarifying these points. I think your responses are reasonable, although I also think the text's discussion of uncertainty should have more nuance to capture these subtle distinctions.
W3: I did not mean to imply that I want to test this with my own prompts (although that would be neat), but I think an interesting application of this method could be understanding uncertainty in unsupervised settings. Can this give us insights about whats going on in a prompt before we collect labels? And sorry, but can you offer a short description of the result and takeaway for the table in djDn?
W4: Thanks for the explanations here. Will you be updating your submission with these changes before the review period ends?
[1] Clark, T. H., Martin, M. B., & Mahowald, K. (2026). Why is AI-generated language so hard to understand? [2] https://claudisms.ai
Dear WKh6, thank you for the concrete pointers and the references.
W1. We share your perspective and appreciate your encouraging words. We already implemented some changes in orange (see paper) to account for LLM-y language and will work on this human-y until the final deadline.
W2. Thank you. Section 3.1 now contains a paragraph that separates the sources of uncertainty (changes in blue), with your coin example and the distinction between TASK and CONTEXT.
W3. Thank you for clarifying. The leave-one-corpus-out evaluation comes close to this setting, as the classifier never sees a prompt of the held-out corpus (or its labels). On the real user prompts of WildChat it still reaches an AUROC of up to 0.82. If we just take a look at our signals, a high entropy at the start of denoising can already support indentifying TASK and UNRESOLVED prompts (Section 5.1). Thus, in a prompt-only setting, such signals could help to assign sources of uncertainty in an unsupervised setting.
The tables in our reply to djDn ask two questions. Does the trajectory carry source information beyond corpus and answer length? And how does it generalize to unseen datasets? We added this ablation in our paper in section D.6 and table 18 (blue). The takeaway is that the trajectory reveals information about the source of uncertainty beyond what we can derive from the prompt origin or length of the answer. The second table shows that this information also transfers to unseen corpora.
W4. Yes, we have uploaded a revised paper with all changes marked in blue (for contents) and orange (for language).
Dear WKh6, thank you for your time and the careful review.
W1. We tried to make the AI usage transparent in our AI use statement. All ideas, experiments and texts are our own. Since none of us are native speakers, we then used LLMs to rephrase it for readability. If you could point us to specific passages, or even suggest phrasings we can build on, we are happy to revise them.
W2(a) Interestingly, this question motivated this paper. While reading the literature, we found several competing views on where the uncertainty of an LLM originates. We collected them from a neutral standpoint and asked whether these human distinctions also appear inside a dLLM.
Two real-world examples of ALEATORIC uncertainty help in answering your question: A WildChat user asks to "write a complex chord progression for a pop song". A CLAMBER prompt asks for "some ideas on where to travel for good sun and beaches". Experts would accept multiple answers to each. A random progression does not make a pop song, and not every place has sun and beaches. However, no lookup and no follow-up question reduces this to a single answer, and this is what ALEATORIC captures.
Your coin example therefore lands in UNRESOLVED or CONTEXT, and we will add it to Section 3.1.
W2(b). You are right that both sources of uncertainty lack information from the user. We follow Kirchhof et al. (2025) in keeping them apart, as the follow-up question differs. For TASK, the prompt does not include an instruction, as in "It's my mom's birthday." For CONTEXT, the task is clear but a detail is missing, as in "When did the first Harry Potter movie come out?", which depends on the country. The dLLMs also treat them differently and the classifiers confuse the two in only 7% to 12% of cases (Table 15).
W2(c) and Q5. We agree that real prompts might mix sources of uncertainty. SOUL labels each source independently to allow for multiple labels. 73 prompts carry two of them. We now trained one classifier head per source. Appendix D.8 of the revised paper reports the results. On ALEATORIC and UNRESOLVED, these mixed prompts score as high as single source prompts, whereas the CONTEXT head drops. As each head outputs its own probability, a prompt with several sources can receive several high scores. A prompt with an unclear source shows up as moderate scores across all heads.
W3. We share this goal, and it motivated the design of SOUL. We combined corpora from uncertainty quantification research with real user prompts. Trained without WildChat, our classifier still recognizes the sources of WildChat prompts (see table in djDn). We made all code available in the anonymous repository to keep the method transparent. It can be adapted as needed for your own prompts. We realize that deploying the dLLMs is a real effort, and we hope you understand that we cannot offer a hosted version with open-ended costs.
W4 and Q1. In its current form, Section 5.1 undersells the quantitative results, which are hidden in the appendix. We will move the key figures to the main text. The interpretable temporal statistics are the named scalars of Table 6, such as the entropy half life, while Figure 3 only illustrates the underlying curves. Section D.1 reports their AUROC, for example 0.79 for the entropy half life on TASK with Nemotron.
Q2. The commit step of position p is tp=min{t∣h[t,p]=1}, the first step after which the position is unmasked and keeps its token. The commit step also highlights what makes dLLMs attractive for our question. Unlike an autoregressive model, a dLLM keeps a full distribution for each position until it commits, so the path to each decision becomes observable. This path is what the posterior cube (see Q3) stores and what our signals measure.
Q3. The cube holds the denoising posterior P[t,p,v]=pθ(yp=v∣x,zt), where x denotes the prompt and zt the partially denoised answer that enters step t. It is exactly the distribution from which the sampler draws. The rates of the sampler are affine in P (Holderrieth and Erives, 2026), so every signal in Table 6 is a chain of aggregations of the cube. We will add this definition to Section 4.1 and revise the notation in Table 6.
Q4. Thank you for pointing this out. τ1/2=min{τ∣H(τ)≤21(H(0)+H(1))} denotes the entropy half life, the time at which the averaged entropy has fallen halfway. We will add this definition to the caption and mark τ1/2 on the axis.