acceptodds
Under review as a conference paper at ICLR 2027

Specification-aware Predictive Uncertainty Decomposition

Abstract

Classical decompositions of predictive uncertainty, such as the information-theoretic one, are defined for a fixed representation of the input object. They do not consider that may be incomplete, nor do they measure how much the missing information can affect the prediction. In Large Language Models, users can clarify an underspecified input without any retraining. The clarified input can describe the user's unobserved intent more accurately and change the predictive uncertainty about the outcome. In this paper, we formalize this setting probabilistically and derive a three-term decomposition of prediction risk into *specification*, residual aleatoric, and epistemic uncertainties. We show that at the original information level, when only is available, the aleatoric uncertainty, represented by a Bayes risk, has two components. The first, which we call specification uncertainty, is the expected reduction in this Bayes risk from revealing the intent. The second is the residual aleatoric uncertainty, i.e., the Bayes risk that remains once the intent is known, averaged over its possible values given . The distinction is important, as specification uncertainty informs the decision to request clarification before making a prediction. We build approximations of these components and connect them to existing uncertainty measures. Our experiments across different domains and modalities show distinct roles for the three components. Specification uncertainty is the most informative for detecting underspecified inputs, while residual aleatoric uncertainty is the most informative for detecting in-distribution prediction errors.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
uncertainty quantificaiton, uncertainty decomposition, specification uncertainty, aleatoric uncertainty, large language models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_D3sCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper considers the problem of quantifying predictive uncertainty in predicting the outcome y given the input object x. More specifically, the authors consider the case where there is a true hidden well-specified intended version of the input, referring to it as z*, and argue that the uncertainty in the predictive outcome given this z* consists of aleatoric and epistemic components. Consequently, the uncertainty of the predictive outcome given x has three components: aleatoric, epistemic (the same as when z* is available, plus an additional term that exists due to under-specification of the input, which we refer to as the specification term, and we propose a method to quantify the three terms. This is particularly relevant for applications where the input can be clarified.

Critical strengths

The problem of uncertainty quantification is becoming more and more important for reliable deployment of ML models, especially the widely used LLMs. The problem of uncertainty quantification given under-specified input is very interesting and quite relevant in this context. I also believe nowadays it is important to acknowledge when a paper is not (or at least does not seem to be) LLM-generated.

Critical weaknesses
  • I still have a hard time digesting how the intent is rigorously defined. How can one simply define z*? 147- suggest it is the fully specified input. What does fully-specified mean? Also, from whose perspective? A user? Maybe user A has a different idea of a well-specification than user B. Is it user-independent? Then it should be defined. If it is not well-defined, how can the specification uncertainty be even quantified?
  • The method is mainly motivated by the context of Large Language Models, where the user providing the input x can be requested to clarify their intent by providing additional information. However, it is not clear what this entire approach means for the case of images or audio, for which the authors provided experiments. What does the intent mean in the context of the image or audio? How can a user clarify them?
  • The LLM/VLM experiments are either focused on specification or the aleatoric uncertainty! This is particularly important, as the authors claim on lines 74-76 that the other baselines approximate epistemic uncertainty based on a single model, which ignores the uncertainty about the model parameters.
  • The third experiments with the audio are very underrepresented and can't be understood easily.
  • What is BALD? Why should it be considered a well-known acronym in the field of UQ?
  • An important yet missing related work is [1] (see section 4.2, for instance).

[1] Gruber et al., Sources of Uncertainty in Supervised Machine Learning – A Statisticians’ View

Questions for authors
  • line 73: However, the outcome can be random even for a fully clarified input, so the remaining uncertainty is not necessarily epistemic.==> can authors provide an example of a fully clarified input for which the outcome is random?
  • how about the discrepancy between p(z∣x)p(z|x) and π(z∣x)\pi(z|x). What kind of uncertainty is that?
3 comments

I still have a hard time digesting how the intent is rigorously defined. How can one simply define z*? 147- suggest it is the fully specified input. What does fully-specified mean? Also, from whose perspective? A user? Maybe user A has a different idea of a well-specification than user B. Is it user-independent? Then it should be defined. If it is not well-defined, how can the specification uncertainty be even quantified?

We consider xx to be a (possibly lossy) representation of z∗z^* The true value of z∗z^* is defined by the data-generating process, which relates xx to yy through z∗z^*. For LLMs, this process is the user, so z∗z^* is indeed a user-specific concept, the complete query that the user has in mind when writing xx. For images and audio, z∗z^* is the complete image or recording. It might be that z∗=xz^* = x, it might not, and the data-generating process is the only source of it. This is exactly why we do not assume that we can recover z∗z^* without requesting additional information, i.e., by asking for clarification. Instead, we consider an approximation of p(z∣x)p(z \mid x) by π(z∣x)\pi(z \mid x), and the behavior of the model under this approximation drives the decision whether we should ask for clarification. Importantly, specification uncertainty does not depend on a particular value of z∗z^*. It depends on the distribution p(z∣x)p(z \mid x) of possible values, given xx. This is natural, since different users may write the same xx with different intents in mind, and p(z∣x)p(z \mid x) represents this variability. Hence, specification uncertainty is well-defined even though the intent is user-specific. Regarding the term "fully specified", in Section 3 we indeed describe the intent as a fully specified input to give an intuition about z∗z^*. However, our derivations do not require full specification. Formally, we only require the conditional independence p(y∣x,z)=p(y∣z)p(y \mid x, z) = p(y \mid z), i.e., once zz is known, xx provides no additional information about yy.

The method is mainly motivated by the context of Large Language Models, where the user providing the input x can be requested to clarify their intent by providing additional information. However, it is not clear what this entire approach means for the case of images or audio, for which the authors provided experiments. What does the intent mean in the context of the image or audio? How can a user clarify them?

In all our specification-related experiments, we are primarily interested in a setup, where xx is the transformation of z∗z^* which carries less information about the label yy. From this point of view, our audio and image experiments are the continuation of what we did for texts. In the text-to-text domain, we model xx by either removing parts of the context related to the question being asked (z∗z^*) or by using the existing ambiguous questions (AmbigQA dataset). We extend this to the multimodal setup. We still use an LLM to answer the questions, but the context is either an image or an audio recording. In the former case, we model lack of information by occluding part of the image with a mask, and in the latter case we mask part of the recording with noise. One could come up with other cases where an input of these kinds can be considered underspecified (in the sense, that it might be possible to improve upon it): a blurry image, an image that crops out the important object, a low-res image with crucial details which are impossible to distinguish. In this case an LLM might ask the user to retake the picture, take a close-up image etc.

The LLM/VLM experiments are either focused on specification or the aleatoric uncertainty! This is particularly important, as the authors claim on lines 74-76 that the other baselines approximate epistemic uncertainty based on a single model, which ignores the uncertainty about the model parameters.

Thank you for pointing to it. First, one important note. One of the main benefits of the decomposition we introduce is that the resulting measures of epistemic uncertainty are canonical, and they are already well-studied prior work [1]. For this reason, in our empirical evaluation we were mostly interested in those new measures (finegrained parts of aleatoric uncertainty), that are novel in a sense of interpretation we provide. Second, those other decompositions we mentioned (in a sense we discussed in Section 3) conflate irreducible uncertainty with epistemic one. What they refer to as “epistemic uncertainty” is “residual aleatoric” in our paper, and can be computed without ensembles, using a single model. Third, we report experiments on OOD detection (a typical proxy for epistemic uncertainty evaluation) in classification setting - see Appendix M, showing that the epistemic component of SEA is empirically valid.

[1] Malinin, A., & Gales, M. (2020). Uncertainty estimation in autoregressive structured prediction. arXiv preprint arXiv:2002.07650.

The third experiments with the audio are very underrepresented and can't be understood easily. We thank the reviewer for feedback and have clarified the presentation of the audio experiments in the final version of the paper - see Section 4.3 and Appendix P.

What is BALD? Why should it be considered a well-known acronym in the field of UQ?

We thank the reviewer for pointing this out and apologize for the use of jargon. By BALD (Bayesian Active Learning by Disagreement) [2], we refer to a classical acquisition function in active learning, derived from information-theoretic considerations. The acquisition function is the conditional mutual information between model parameters and target labels and is ubiquitously used as a measure of epistemic uncertainty in the literature. The way it is introduced also allows recovery of total and aleatoric uncertainty (and epistemic, which is defined by BALD itself). Strictly speaking, only the epistemic component is denoted as BALD in the original paper, used as an acquisition function in the active learning setup. We agree that referring to the whole decomposition as such is not particularly precise. We have made this distinction and convention explicit in the updated version. Thanks for bringing this to our attention.

[2] Houlsby, N., Huszár, F., Ghahramani, Z., & Lengyel, M. (2011). Bayesian active learning for classification and preference learning. arXiv preprint arXiv:1112.5745.

An important yet missing related work is [1] (see section 4.2, for instance).

Thanks for bringing this paper to our attention, as we believe it is a great counterpart to our work. We agree with Gruber et al. that what we call “specification uncertainty”, and they refer to as “data-model uncertainty”, primarily affects the aleatoric component. They assume a very similar data model, and make the same assumption on conditional independence between XX and YY given ZZ. Our main difference here, is that they define aleatoric uncertainty as the variance of the Y∣XY \mid X distribution, and obtain a decomposition of it into expectation of variance and variance of expectation. We believe that this particular form is, in fact, a specific case of SEA decomposition, if one would assume MSE loss (proper, but not strictly proper s.r.), instead of log-loss, as we did in our experiments.

We also agree on the estimability of epistemic uncertainty with the first-order models. They claim, and we agree, that proper estimation of epistemic uncertainty requires a posterior distribution over model parameters, hence our focus on ensembling in experiments. It is important to stress that ensembles are not an essential part of SEA, but rather a particular choice of a second-order approximation, and other approximations exist.

We have therefore added the reference to the updated version of the paper, and a proper discussion on another special case of SEA.

line 73: However, the outcome can be random even for a fully clarified input, so the remaining uncertainty is not necessarily epistemic.==> can authors provide an example of a fully clarified input for which the outcome is random?

Yes, sure!

One narrow type of such questions concerns properties of the world that are, in principle, unpredictable. For example, the question "Will this radioactive atom decay within the next hour?" cannot be answered deterministically because of the inherent randomness of quantum mechanics.

A broader and more practical type of question is one whose answer is not determined by the information available at prediction time, even though the question itself is fully specified. For example, "Will it rain in London on December 1, 2026?" is unambiguous, but its answer is random given the information available today. Adding more details to the question does not change this. Weather is known to be unpredictable more than about two weeks ahead, since small errors in the measured initial state of the atmosphere grow exponentially over time [3]. Similarly, the outcome is random for questions with several valid answers (e.g., "Suggest a name for a new coffee shop"). In our decomposition, this randomness corresponds to the residual aleatoric term. Neither clarification nor model improvement can reduce it, so treating it as epistemic would suggest refining a predictor that may already be perfect.

[3] Lorenz, E. N. (2017). Deterministic nonperiodic flow. In Universality in Chaos, 2nd edition (pp. 367-378). Routledge.

how about the discrepancy between p(z∣x)p(z \mid x) and π(z∣x)\pi(z \mid x) . What kind of uncertainty is that?

Thank you for the interesting question!

As we define in line 234, our predictor is q=EZ,Θ[qZ,Θ]q = E_{Z,\Theta}[q_{Z,\Theta}], where the expectation is taken under π(θ,z∣x,D)\pi(\theta, z \mid x, D). Hence, any discrepancy between π(z∣x)\pi(z \mid x) and p(z∣x)p(z \mid x) results in a discrepancy between qq and pp, which is taken into account in the epistemic term DG(p∥q)D_G(p \| q) of Eq. 3. We want to emphasize that this agrees with the canonical definition that the lack of knowledge about the ground-truth process pp is epistemic uncertainty. In contrast, in the single-model decompositions we discussed (Hou et al., 2024, Walha et al., 2026), the excess risk is not part of the decomposition (Eq. 6), so their terms do not capture this discrepancy.

We also use π(z∣x)\pi(z \mid x) as a plug-in estimate of p(z∣x)p(z \mid x) when computing the components in Eq. 4. For this part, the discrepancy is not taken into account and should instead be considered an estimation error. It affects all three estimates, particularly those of the specification and residual aleatoric terms. The coarser the plug-in estimate, the less accurate they are.

Reviewer_66k4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces a three-term decomposition of predictive uncertainty into Specification, Epistemic, and residual Aleatoric uncertainty (SEA). Specification uncertainty measures the expected reduction in Bayes risk from revealing missing input information. The authors derive Bayesian approximations and evaluate the components on text, image, and audio tasks, showing that specification uncertainty is particularly effective at detecting underspecified inputs.

Critical strengths
  1. The motivation for decomposing specification uncertainty is reasonable. Unlike conventional uncertainty decompositions, the proposed framework explicitly considers uncertainty that could be reduced by clarifying the input. This distinction is meaningful for interactive AI systems.

  2. The experiments across text, image, and audio show that specification uncertainty is more informative than conventional uncertainty components for detecting underspecified inputs.

Critical weaknesses

The main issue is the disconnection between theory and empirical evaluation. The paper defines specification uncertainty as the expected reduction in Bayes risk after revealing the true intent z⋆z^{\star}, motivating it as a signal for deciding when to request clarification. However, the experiments primarily evaluate whether this uncertainty score can detect underspecified inputs. They did not directly validate its practical utility. It remains unclear whether higher specification uncertainty actually correlates with a larger performance gain after clarification, or whether it leads to better clarification decisions compared to simpler baselines. Moreover, the theoretical framework assumes access to the true intent distribution p(z∗∣x)p(z^*|x), whereas the practical estimator relies on LLM-generated clarifications. The authors did not sufficiently justify or evaluate whether these generated clarifications faithfully capture the underlying intent distribution, leaving a critical gap between the theory and its implementation.

Questions for authors
  1. Does higher specification uncertainty actually correspond to greater prediction improvement after clarification? Can this be verified using the original inputs or ground-truth clarifications?

  2. Does specification uncertainty lead to better clarification decisions than uncertainty-based baselines?

1 comment

The paper defines specification uncertainty as the expected reduction in Bayes risk after revealing the true intent , motivating it as a signal for deciding when to request clarification. However, the experiments primarily evaluate whether this uncertainty score can detect underspecified inputs. They did not directly validate its practical utility. It remains unclear whether higher specification uncertainty actually correlates with a larger performance gain after clarification

We agree that connection between detection performance and downstream accuracy gains were too implicit in the original version of the paper. We reported the detection performance, as well as catastrophic accuracy degradation when models are evaluated on underspecified inputs, but no direct connection was made. To remedy this, we added a new experiment, where a mix of underspecified and well-specified inputs is processed by a two-stage pipeline, first clarifying and then rejecting at select budgets, and reporting the resulting accuracy. New experiments are presented in Table 2 of the updated paper, and a more detailed investigation is added as a new Appendix D. In all considered scenarios, SEA results in better eventual outcomes as measured by predictive accuracy compared to the same set of baselines, directly connecting dedicated underspecification detection capability with downstream performance gains.

or whether it leads to better clarification decisions compared to simpler baselines

We agree that selection of baselines can be widened, and we added two simple baselines that could be readily computed from our existing data to all experiments, with SEA maintaining its edge in all considered setups.

Moreover, the theoretical framework assumes access to the true intent distribution p(z^* | x), whereas the practical estimator relies on LLM-generated clarifications. The authors did not sufficiently justify or evaluate whether these generated clarifications faithfully capture the underlying intent distribution, leaving a critical gap between the theory and its implementation.

We agree with the reviewer that the true intent distribution p(z∣x)p(z \mid x) is unknown in practice. Therefore, it has to be approximated. Note, however, that it is not a problem of our approach per se. As we discuss in Section 3.2, the pairs (x,y)(x, y) can only determine p(y∣x)p(y \mid x), but not how its Bayes risk is split between specification and residual aleatoric uncertainty. Therefore, any estimate of this split, including those of Hou et al. (2024) and Walha et al. (2026), uses an additional model of the intents. Ideally, this model would be learned jointly with the predictor from the same training data, as in the general form π(θ,z∣x,D)\pi(\theta, z \mid x, D) of our Bayesian model. But for LLMs, this is impractically expensive. We therefore follow prior work (Hou et al., 2024, Walha et al., 2026) and use large pretrained models as a practical approximation of p(z∣x)p(z \mid x) and discuss possible limitations in Appendix C. Nevertheless, empirically, we see that this choice of approximation works well intuitively and is interpretable. Specification uncertainty is the best detector of underspecified inputs for each domain we considered. Finally, we see our main contribution in the decomposition. It separates specification, residual aleatoric, and epistemic uncertainty and generalizes several existing decompositions. Any better approximation of p(z∣x)p(z \mid x) can be used in our framework without changing it.

Does higher specification uncertainty actually correspond to greater prediction improvement after clarification? Can this be verified using the original inputs or ground-truth clarifications?

Yes, see Table 2 and Appendix D in the updated manuscript.

Does specification uncertainty lead to better clarification decisions than uncertainty-based baselines?

Yes, see results with new baselines in all experimental tables.

Reviewer_PRV9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SEA, an uncertainty decomposition framework for LLM. SEA introduces specification uncertainty in addition to the standard aleatoric and epistemic uncertainty, and specification uncertainty is reducible given clarification of user's intent. SEA was evaluated across QA tasks and image classification tasks and the three terms (specification, aleatoric, epistemic uncertainty) have been shown to play distinct roles. In particular, specification uncertainty is shown to be the most informative for detecting underspecified inputs.

Critical strengths
  1. The motivation is clear and the newly introduced specification uncertainty seems very natural to me in LLM context as we know the uncertainty of LLM can be reduced with intent clarification.
  2. The paper extends the standard KL based uncertainty decomposition framework to incorporate Bayes risk as a more general setup.
  3. The experiments are conducted across a diverse set of modalities.
  4. The manuscript is clear and easy to follow.
Critical weaknesses
  1. Does the assumption in Eq 1 make sense? It seems to assume we can get a fully specified z from x in one-shot by LLM. In practice, there can be multiple z corresponding to one same intent, and there can also be partially specified intent.
  2. SEA requires multiple LLMs to form π(θ,z∣x,D)\pi(\theta, z | x, D), which is computationally more expensive than UQ decomposition method that only requires a single LLM.
  3. SEA seems to perform worse tha BALD for text QA error-detection (Table 2).
Questions for authors

It would be interesting to discuss [1] and contrast SEA with it, which also considers Bayesian UQ decomposition for LLM, but doesn't rely on ensemble of multiple LLMs.

[1] Jayasekera et al.. Variational Uncertainty Decomposition for In-Context Learning. NeurIPS 2025

3 comments

Does the assumption in Eq 1 make sense? It seems to assume we can get a fully specified z from x in one-shot by LLM. In practice, there can be multiple z corresponding to one same intent, and there can also be partially specified intent.

Yes, the assumption in Eq. 1 makes sense. The only assumption in Eq. 1 is the conditional independence p(y∣x,z)=p(y∣z)p(y \mid x, z) = p(y \mid z). Informally, zz is at least as informative about yy as xx, so once zz is known, xx provides no additional information about yy. The setting the reviewer describes, with multiple plausible zz for the same xx, is exactly the one we consider. Let us clarify what Eq. 1 does not assume.

Eq. 1 does not assume that a single fully specified zz can be obtained from xx. For a given xx, written by a user with a certain intent z∗z^* in mind, there are indeed multiple plausible values of zz. We describe them by the distribution p(z∣x)p(z \mid x), which we keep throughout the derivations and do not assume to be degenerate.

Eq. 1 is also not tied to LLMs. It describes the ground-truth process, and an LLM appears only later, as one of several possible ways to approximate p(z∣x)byπ(z∣x)p(z \mid x) by \pi(z \mid x). For example, in our VQS experiments, we do not use an LLM at all and obtain zz with an inpainting model instead.

The reviewer's point on partially specified intent is a very good one. In fact, our conditional independence assumption does not require zz to be fully specified. For example, if a clarification resolves only part of the ambiguity, we can take z to be x together with this partial clarification. Then p(y∣x,z)=p(y∣z)p(y \mid x, z) = p(y \mid z) holds by construction, and the specification term measures the expected reduction in Bayes risk from this partial clarification.

We will revise the passage in Section 3 to make this clearer. We also note, that this assumption has been previously considered in the relevant literature, see for example [1] (4.2.3)

[1] Gruber et al., Sources of Uncertainty in Supervised Machine Learning – A Statisticians’ View

SEA requires multiple LLMs to form π(θ,z∣x,D)\pi(\theta, z \mid x, D), which is computationally more expensive than UQ decomposition methods that only require a single LLM.

We agree with the reviewer that π(θ∣D)\pi(\theta \mid D) is a distribution over the parameter space, and hence estimating the epistemic term requires samples from it (e.g., with an ensemble of models). There are several points we would like to clarify.

First, single-model decompositions do not estimate uncertainty about the ground-truth process. Hou et al. (2024) and Walha et al. (2026) interpret the expected uncertainty remaining after clarification (term (ii) in Eq. 6) as epistemic uncertainty. But canonically, epistemic uncertainty should signal points where the prediction can be improved by refining the model at hand. However, even for the ideal predictor, with π(z∣x)=p(z∣x)\pi(z \mid x) = p(z \mid x) and qz=pzq_z = p_z, this term equals −EZ[G(pZ)]-\mathbb{E}_Z[G(p_Z)], which is positive whenever the outcome is random given the intent (see our discussion in Section 3.2). Under their interpretation, the ideal predictor would therefore still have positive epistemic uncertainty. For example, the question "Will this radioactive atom decay within the next hour?" is fully specified, but its correct answer is random by the laws of physics, and no refinement of the model can remove this uncertainty. In SEA, this term is the residual aleatoric uncertainty, i.e., a part of aleatoric uncertainty, while epistemic uncertainty is a separate term that requires samples from π(θ∣D)\pi(\theta \mid D).

Second, the specification and residual aleatoric terms do not require an ensemble. With a single model, the epistemic term vanishes, and SEA reduces to Eq. 6, which has the same computational cost as the single-model decompositions of Hou et al. (2024) and Walha et al. (2026) (see references in our paper).

Third, epistemic uncertainty is canonically defined as a lack of knowledge about the ground-truth data-generating process. However, this process is never known in practice (especially for problems involving LLMs), and hence one needs to approximate it. Ensembles are among the most accurate ways to do so [1], yet, as the reviewer notes, they are expensive. We chose this approximation to verify our intuition about the interpretation of components as reliably as possible. Simplifications (e.g., assuming that a specific model approximates the ground truth well enough) may be inaccurate and distort the decomposition (and hence interpretation). At the same time, our framework does not depend on this particular choice, and any other approximation of π(θ∣D)\pi(\theta \mid D) can be used in Eq. 4.

Finally, we have a dedicated section (Appendix I) in which we estimate the terms without ensembles. Specifically, in Tables 7 and 8, we evaluate each member of our heterogeneous ensemble separately. For every model and on every text and visual QA dataset, specification uncertainty is the best detector of underspecified inputs. Thus, if epistemic uncertainty is not needed, SEA can be used with a single LLM.

[1] Izmailov, P., Vikram, S., Hoffman, M. D., & Wilson, A. G. (2021). What are Bayesian neural network posteriors really like? In International Conference on Machine Learning (pp. 4629–4640). PMLR.

SEA seems to perform worse tha BALD for text QA error-detection (Table 2).

Our idea behind the experiment was to show that the residual aleatoric uncertainty component of SEA decomposition provides a better signal for error detection than specification uncertainty. This is what Table 2 indeed illustrates.

We agree, however, that the point "for error detection, residual aleatoric is always better than whole aleatoric" is not supported by this table. Specifically, BALD aleatoric computed over the clarified posterior corresponds to the sum of SEA specification and SEA residual exactly. Therefore, we have renamed it to “SEA_{res + alea}” in the Table.

We note that the setup behind Table 2 is very limited - the dataset only contains well-specified, in-distribution examples, so the fact that BALD provides better error detection AUROC in such a setting is not an argument against the usefulness of SEA. To better highlight the usefulness of three-part decomposition, we conducted an additional two-stage experiment on the data that contains both underspecified and well-specified inputs, where we first clarify and then reject inputs. We moved the discussion about specificity of SEA components to ID errors to Appendix C, and new Table 2 reports the result of this new experiment. We kindly ask the reviewer to consider these new results as well.

It would be interesting to discuss [1] and contrast SEA with it, which also considers Bayesian UQ decomposition for LLM, but doesn't rely on ensemble of multiple LLMs. [1] Jayasekera et al.. Variational Uncertainty Decomposition for In-Context Learning. NeurIPS 2025

We thank the reviewer for this addition to our related work. Indeed, both SEA and VUD employ augmented inputs (clarifications in SEA case, fantasized continuations in VUD), however the overall approach is quite different. VUD considers the LLM itself to be a Bayesian model, in the sense that it updates its predictions according to Bayes rule, given in-context examples, and produces bounds on aleatoric and epistemic uncertainties at a given input. So VUD does not produce estimators of these components, but rather bounds that rely on assumption of bayesian updates, and additivity is maintained by construction. Conversely, SEA gives a principled additive decomposition of uncertainty (in fact, a family thereof). Most importantly, SEA is a three-part decomposition, conserving both aleatoric and epistemic terms, while adding a dedicated component for specification uncertainty. Practically, VUD indeed does not rely on ensembling in parameter space, at inference time it relies on sampling of fantasized continuations and enumeration over label space. This brings practical inference costs of VUD on par with those of SEA, but it does not require fine tuning of models. However, as shown by our experiments with heterogeneous pre-trained ensemble, one is not required to train ensemble of their own.

Comments on the paper

1 comment

We thank all the reviewers for their insightful comments and suggestions, which prompted much discussion among the authors and a deeper understanding of the findings, as well as valuable improvements in experimental section. However, we would like to note that the focus of the paper as we see it is not on the individual empirical performance of the SEA decomposition components. We feel that the substantial theoretical results presented in the paper attracted less attention from the reviewers, compared to empirical considerations. We would like to highlight these theoretical results here. Specifically, we have shown that:

  1. The three-part additive decomposition that cleanly separates specification uncertainty from aleatoric and epistemic components does exist
  2. This decomposition is principled and can be derived under very basic assumptions about the underlying data model. These assumptions are standard and were previously considered in the literature (see 4.2.3 in [1])
  3. It settles the tension in the recent literature on specification uncertainty regarding the nature of it as aleatoric or epistemic. SEA clearly shows that at the information level considered in the paper, specification uncertainty is a particular (and extractable) part of aleatoric uncertainty
  4. It turns out to be a strict generalization of several existing decompositions ([1], [2], [3], [4]), both classical and recently proposed, and reduces to these decompositions for particular choices of losses, approximation strategies and assumed data distributions
  5. Additional experiments show that it performs best specifically when used as a driver of a multi-step decision-making procedure, validating the need for separate accounting of specification, aleatoric and epistemic uncertainties as decision-making signals.

We hope that this will be taken into account during the private discussion period, and will be reflected in the final scores.

[1] Gruber et al., Sources of Uncertainty in Supervised Machine Learning – A Statisticians’ View

[2] Hou et al., Decomposing uncertainty for large language models through input clarification ensembling

[3] Walha et al., Fine-grained uncertainty decomposition in large language models: A spectral approach

[4] Houlsby et al., Bayesian active learning for classification and preference learning

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.