- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes W-GBDL, a Bayesian last-layer method for uncertainty quantification that extends the Bayesian non-negative decision layer (BNDL). On top of a deterministic backbone, each input is mapped to a posterior over non-negative latent factors, and each class is represented by a learnable reference distribution. Both use Gamma priors and diagonal Weibull variational posteriors. The input posterior is compared to each class reference with the 2-Wasserstein distance, which has a closed form for diagonal Weibulls, and the closest class is predicted. The distances define Dirichlet concentrations and the uncertainty score is the normalized entropy of the Dirichlet mean. Training combines a Gumbel-Softmax-relaxed cross-entropy, KL terms toward the Gamma priors, and an EDL-style Dirichlet regularizer. Input-side and class-side dispersion are interpreted as aleatoric- and epistemic-related uncertainty. Experiments with ResNet-50 on CIFAR-10/100 cover accuracy, calibration, misclassification and OOD detection, and noisy inputs. Experiments with LLaMA3-8B fine-tuned on six multiple-choice QA benchmarks cover calibration. Ablations cover the posterior components, the KL terms, the distribution family and the temperature.
Critical strengths
Papers main strength are:
-
Consistent empirical gains at single-pass cost, across two modalities. OOD detection: on CIFAR-10/ResNet-50, W-GBDL has the best mean OOD AUPR among single-model methods. Its run-to-run variance is also much lower. It matches a 5-member deep ensemble (85.92) at roughly single-forward-pass inference cost, and combining it with an ensemble improves further, so it complements ensembling rather than replacing it.
-
A tractable closed-form construction. Two pieces make every training term analytic: the Wasserstein distance between diagonal Weibull distributions in closed form (Eq. 11, via 1-D quantile matching), and the analytic Weibull–Gamma KL. No Monte Carlo transport estimation or posterior sampling is needed. That makes distribution-to-distribution matching cheap enough.
This is what allows the approach to scale. It is a modest but solid technical contribution that others could reuse for distance-based uncertainty layers.
The method is a drop-in last layer that improves uncertainty estimates without sampling or multiple models, on both a vision backbone and an LLM.
Critical weaknesses
The paper argues that reliable uncertainty must distinguish predictive confidence from input–class compatibility. However, the score U_W (Eq. 15) is exactly the normalized entropy of softmax, which does not change when all distances d_ic shift by the same constant. An input uniformly far from every class reference receives the same uncertainty as one uniformly close to them, so absolute compatibility is discarded and only relative class preference remains. Similarly, tr(Σ_θ,i) cancels from U_W, and Eq. 14 is a generic identity valid for any coupling, in which dispersion enters only as a mismatch in spread. The aleatoric/epistemic interpretation in Sec. 3.4 is therefore unsupported. The paper’s conceptual contribution rests on a property the score provably lacks. The observed OOD gains are thus unexplained by the stated mechanism.
Questions for authors
1. Why Gamma priors and non-negative latents? Gamma priors and non-negative latents seem to be inherited from BNDL rather than required by W-GBDL. The likelihood is Dirichlet–Categorical, so conjugacy plays no role. with squared-Euclidean cost does not need positive support, and the features are already non-negative via Softplus. Could the authors explain why positivity of and is essential to the method?
2. Why not other distribution families? Several families give both closed-form (for diagonal factorizations) and analytic KL. Examples are diagonal Gaussians, and log-normals: with the comonotone coupling, the cross term is . Log-normals are also positive and skewed. Have the authors tried such alternatives? If Gamma–Weibull is specifically needed, could you justify why exactly ?
3. Is the Weibull only there for closed-form inference? Sec. 3.2 justifies the Weibull by its similarity to the Gamma and by the analytic Weibull–Gamma KL. Both arguments apply equally to a Gamma posterior, whose KL to a Gamma prior is also analytic and which is reparameterizable via implicit gradients. Is the Weibull chosen mainly because its closed-form quantile function gives the analytic in Eq. (11) ? If so, please state this explicitly.
4. Is the aleatoric/epistemic interpretation justified? Sec. 3.4 links input-side dispersion to aleatoric uncertainty and class-reference dispersion to epistemic uncertainty. Could the authors justify this formally? Several observations argue against it:
- Eq. (14) holds for any coupling of any two distributions. In it, dispersion enters only as a mismatch in spread, so a wide input distribution matched to an equally wide reference adds no cost.
- is the same for every class and cancels in .
- The class-side variance is global, so it cannot grow for inputs far from the training data, unlike typical epistemic uncertainty.
5. Why is the Gumbel-Softmax relaxation needed? Gumbel-Softmax is meant for backpropagating through samples of an unobserved discrete variable. In W-GBDL the label is observed, and the exact likelihood (Eq. (6)) is already closed-form and differentiable in all parameters. Could the authors explain why the relaxation in Eqs. (10)–(12) is needed? As written, the relaxation s a biased, noisy surrogate of the exact cross-entropy.
6 comments
Q3: Is the Weibull only there for closed-form inference?
The main computational reason for choosing Weibull is its explicit quantile function, which gives the analytic Wasserstein cost in Eq. (11). We agree that similarity to Gamma and an analytic KL alone do not distinguish this choice from a Gamma posterior.
For a Weibull variable, Substituting this into the one-dimensional quantile formula and setting gives Together with the second moments, this yields the closed-form transport cost for arbitrary positive Weibull shape and scale parameters.
A Gamma posterior also has an analytic Gamma--Gamma KL; therefore, KL tractability alone does not justify choosing Weibull. The remaining difficulty is the transport cross term. For in rate parameterization, the quantile is , where is the regularized lower incomplete Gamma function. The transport cross term therefore involves For general unequal shapes, this does not simplify through the Gamma-function identity used for Weibull; direct quantile evaluation generally requires numerical inversion and quadrature. Implicit reparameterization enables differentiation through Gamma samples but does not itself provide an analytic transport cost. Special cases remain tractable: if , then
Thus, Weibull provides positive latent variables together with both analytic Wasserstein matching and analytic KL regularization against the Gamma prior. We have added a concise explanation of this choice immediately after the closed-form Weibull derivation in Appendix B.1.
Q4: Is the aleatoric/epistemic interpretation justified?
The concerns raised here are consistent with the qualification in Reviewer o6an Q3, but we further clarify their mathematical implications. Equation (14) is valid for any coupling and should not be interpreted as an identifiable decomposition of uncertainty. A broad input distribution does not necessarily incur an additional spread-mismatch cost when the matched class-reference distribution has a comparable spread, although a mismatch in their means may remain.
For a fixed input, the explicit local covariance term is shared across all classes. Since the class probabilities are obtained by normalizing the class-wise transport costs, this class-independent contribution cancels from the predictive entropy. Any remaining effect of the local dispersion can only arise indirectly through the class-dependent optimal-coupling term.
The class-reference covariance is learned globally and shared across inputs. It can affect the compatibility cost for a class, but it cannot increase as a function of how far a particular test input lies from the training data. We therefore do not interpret it as an input-dependent epistemic uncertainty estimate.
These observations further clarify the distinct modeling roles of the two variational factors. The local factor is input-conditioned, whereas the class-reference factor is learned globally and shared across inputs. Their dispersions therefore influence the transport cost through different pathways, as discussed in Reviewer o6an Q3. We will make this distinction clearer in Sec. 3.4.
Q5: Why is the Gumbel-Softmax relaxation needed?
The Dirichlet–Categorical hierarchy and Gumbel–Softmax serve connected roles in our stochastic decision formulation. Following Eq. (6), we analytically marginalize the Dirichlet variable, obtaining the categorical label distribution Gumbel–Softmax then provides a continuous stochastic relaxation of this marginalized categorical decision.
During training, we draw one independent Gumbel vector per input and optimization step to estimate the expectation in Eq. (12). This objective trains the decision layer over randomly perturbed soft class assignments, with gradients propagated through their continuous relaxation. Thus, the purpose of Gumbel–Softmax is to introduce stochasticity into decision training, rather than merely to make the predictive probabilities differentiable. At test time, predictions and predictive entropy are computed directly from , without Dirichlet or Gumbel sampling.
The exact observed-label cross-entropy, , provides a direct categorical alternative. We compare this alternative with our full hierarchical stochastic formulation in the table provided in our response to Reviewer o6an's W1, removing the Dirichlet KL and Gumbel–Softmax training step together while retaining the Wasserstein matching and local/global latent regularization. The full formulation improves mean accuracy from 92.70% to 93.11%, reduces ECE from 1.58% to 1.31%, and lowers NLL from 0.25 to 0.23. These results support the joint benefit of distributional regularization and stochastic decision training. We will clarify that Eq. (12) defines this stochastic training surrogate, while Eq. (6) supplies the analytically marginalized predictive probabilities.
We sincerely appreciate the valuable suggestions you have provided for our paper. Your feedback has played a crucial role, offering constructive and detailed suggestions for enhancing the overall framework of the paper.
Our choice follows a joint modeling and computational consideration: the latent representation should describe positive activation intensities, and the resulting distributions should support efficient regularization and distributional matching. As discussed in our responses to Reviewer D4UX W1/W3, these considerations motivate the Gamma–Weibull specification.
Non-negative latent representation. We interpret each coordinate of θi and ϕc as the strength of a latent feature, with values near zero corresponding to weak or inactive factors. Modeling both sides on the positive domain gives the input and class reference a consistent activation-based interpretation. Softplus and positive-support distributions serve complementary purposes: Softplus constrains the deterministic features or parameters, while the distribution family constrains the support of the random latent activations. A positive Gaussian mean, for example, does not prevent negative latent realizations. Thus, positive-support distributions are needed to preserve this interpretation throughout the probabilistic representation.
Joint choice of the prior and variational family. The Gamma prior directly regularizes positive activation intensities. In the reported experiments, Gamma(1,1) is an exponential prior whose density decreases with activation magnitude, favoring smaller intensities while permitting larger activations when supported by the supervised objective. The Weibull family provides flexible positive distributions, including densities concentrated near zero and skewed unimodal shapes. Combined with the Gamma prior, it gives an analytic KL term; its explicit quantile function additionally yields the closed-form Wasserstein cost in Eq. (11). This combination allows both posterior regularization and distributional matching to be evaluated analytically.
Differences from alternative families. A diagonal Gaussian offers analytic computation but does not preserve the positive activation support. A log-normal preserves positivity and skewness, but its density approaches zero at the origin, whereas Weibull distributions can represent substantial near-zero density through their shape parameter. This provides a distinct inductive bias for representing weak or inactive factors alongside strong activations.
We tested these alternatives in the distribution-family ablation reported in the newly added Table 9 in the Appendix, which further extends the scope of our ablation study, as discussed in our response to Q2 from Reviewer o6an. The Gamma–Weibull configuration achieves 86.24% OOD AUPR, compared with 80.58% for the Gaussian configuration and 83.14% for the log-normal configuration. These results provide empirical support for the proposed combination. Although the non-negative latent formulation builds on BNDL, W-GBDL uses the resulting distributions on both sides of an explicit Wasserstein comparison. Our choice therefore connects the activation-based representation with tractable distribution-to-distribution matching, which is the central design of our decision layer.