acceptodds
Withdrawn from ICLR 2027

From distribution to sample: Quantifying Uncertainty for Heterogeneous Distributions

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Reliable uncertainty quantification (UQ) becomes particularly challenging under heterogeneous distributions. In practice, it can be often observed that many UQ methods exhibit calibration degradation as the proportion of heterogeneous samples increases. We argue that this limitation arises mainly because covariate shift and out-of-distribution (OOD) are only defined with regard to distributions, failing to provide a direct criterion for assessing individual inputs. To address this, we propose a sample-based lightweight plug-in named Dual-signal Coverage-aware UQ (D-CUQ). D-CUQ leverages two key metrics: the Manifold-deviation Signal, capturing where an input lies relative to the ID manifold, and the Coverage-gap Signal, capturing whether the local tangent geometry around it is spanned by training data. We further establish a formal connection between the two levels, showing that distribution-level quantities arise as aggregates of their sample-level counterparts. Extensive experiments on classification and regression benchmarks demonstrate that D-CUQ significantly improves uncertainty estimation for heterogeneous test distributions.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
uncertainty quantification, heterogeneous test distribution

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_gr4kOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper addresses calibration degradation of uncertainty quantification (UQ) under heterogeneous mixed distributions, where in‑distribution, covariate‑shifted and semantic OOD samples are mixed together. It proposes a lightweight plug‑in module D‑CUQ built on frozen pre‑trained model features. Two signals are computed: manifold‑deviation signal measures positional offset from ID manifold, and coverage‑gap signal evaluates local geometric coverage given training data. The paper builds theoretical connection between sample‑level signals and distribution‑level shift notions, and designs an entropy‑guaranteed calibaration strategy. It works with existing UQ baselines without changing classification labels or regression prediction mean. Experiments and ablations on classification and regression tasks verify the joint improvement brought by dual‑signal.

Critical strengths
  1. The ablation suite is carefully designed to exclude key alternative explanations. The shuffled‑dual control proves performance gain roots in sample‑wise paired dual‑signal, instead of mere output smoothing. Comparisons against single‑signal and global scaling further confirm both signals are necessary. These empirical evidences well support the validity of this plug‑in method.
  2. This work implements both classification and regression to verify the generality of feature‑based dual‑signal. It works as plugin on top of frozen backbone without retraining, compatible with multiple mainstream UQ baselines. For regression task, target binning adaptation makes dual‑signal mechanism function without relying on specific output head, which broadens its applicable scope.
Critical weaknesses
  1. The key solution is incremental combinatorial innovation, and its theoretical correspondence only holds for one‑way operational property. Underlying operators are from prior works (Lee et al.,2018; Mukhoti et al.,2023). The sample‑distribution relation has counter‑examples and cannot equal true sample shift generation type. The UQ degradation under mixed distribution is observed in Ovadia et al. (2019); full‑spectrum OOD concept from Yang et al. (2023), OpenOOD v1.5 benchmark from Zhang et al. (2024).
  2. Evaluations reply on self‑constructed heterogeneous mixed test set, and the method has relatively narrow applicable scenarios. There exists no community‑standard QmixQ_{mix} benchmark, raising barrier for fair cross‑method comparison. As post‑processing plug‑in, it cannot fix representation defects and heavily relies on feature quality from frozen model, only valid under constrained setup.
Questions for authors
  1. Ablation only proves signal utility. How to empirically verify signal correspondence with two types of shift generation mechanism?
  2. Why there is no quantitative experiment comparing against Geometric‑ODIN which also uses dual scores for shift characterization?
  3. Could you provide quantitative results of D‑CUQ when backbone representation quality degrades?
  4. Under your operational definition, what is approximate fraction of counter‑examples in real test?
Reviewer_2hMJOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper introduces a calibration plugin for classification and regression backbones based on the distance of an instance from the manifold of the data points in the feature space. Calibration is very important in high-stakes decision making. The proposed method also appears to improve prediction accuracy in addition to calibration metrics.

I find the paper and the source code/supplement very hard to follow. I list my issues in Questions.

The results could make a very good paper but in its present form lots of polishing is needed beyond the timeline of ICLR reviewing.

Critical strengths

Calibration is very important in high stakes decision making and is a hard problem. With careful additional explanations (see Weaknesses and Questions) the results could make a strong paper.

Critical weaknesses

W1 I list in Questions quite a few issues I came across while trying to understand the paper. Major polishing is needed beyond the timeline of ICLR reviewing, also for the source code.

W2 TabPFN is an important baseline/backbone that should be included in the experiments.

W3 While very nice, also very surprising that the algorithm improves both calibration and accuracy metrics, an explanation, ablation, additional experiments would make the contribution much stronger.

Questions for authors

Please summarize the notation. In my understanding z/Z is an embedding of x but it is never stated. Notions in eq (2) are defined only later in eq (3) and in fact U_S(x) is never defined although I can (hopefully) guess the meaning. Eq (4) is very hard to read due to spacing and I don't see where is x in the formula.

Some descriptions are too vague to understand, for example L165 "pointing along one of them" and L166 "standardized reference representations" -> of x? what is standardized?

In Section 3.3 a clustering into k partitions is used but only implicitly defined. I wonder if an actual clustering algorithm is involved? It seems to be. Which algorithm, and how sensitive are the results to k and the algorithm itself?

In Algorithm 1, I do not see how is OOD and covariate shift separated? By the Introduction this is the main contribution but I would need more explanation. L312 reference to Appendix seems to be broken, which section?

In Table 1, EDL is one backbone but EDL is never defined. My guess Ensemble Deep Learning, but there is also Deep Ensemble, with proper reference.

Also in Table 1 I am puzzled how a calibration heuristic improves accuracy (AUROC, AURC)? How are training, testing, and calibration sets selected? Can you ablate on the source of accuracy improvement?

In Section 5 Discussion, the section seems to be about an extra experiment and not discussion of the previous ones. The flow of the text is a bit broken here. Also what does in L441 "they are not confined to it" mean?

Tabular foundation models, in particular TabPFN is considered to be state of the art and in my experience perform exceptionally well for calibration too. TabPFN should be included as baseline/backbone.

Regarding the source code, I did not attempt to run it, just to understand the structure. It was not fully clear to me which experiments, with what parameters, need to be run to reproduce the paper. The main Readme.md is not sufficiently detailed and I got confused when looking at subcomponents. The root directory has a very large amount of files, many of them are non-english, I wonder which of them are needed.

ECE is used as calibration metric. ECE is known to be inconsistent and also dependent on binning. SKCE Widmann et al. (2021) is recommended to be used, I assume there should be no computational limitations for the not very large problems considered.

L084: how do you compute AUROC for regression? Is it for OOD?

Reviewer_VjKXOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes D-CUQ, a sample-based plug-in method designed to improve calibration performance under heterogeneous distributions characterized by a combination of covariance shift and OOD data. The method integrates two signals. One based on the distance from the ID manifold, and the other indicating whether a given sample lies within the space spanned by the local tangent geometry of the training data. In the experiments, the proposed method was evaluated using various benchmark datasets across both classification and regression tasks, and its effectiveness was confirmed across several model architectures.

Critical strengths

None.

Critical weaknesses

I evaluated the paper primarily in terms of the theoretical and empirical justification of its core framework. I identified two major weaknesses that significantly affect my overall assessment.

1. Insufficient Justification and Theoretical Connection in the Problem Formulation

The paper lacks a sufficiently rigorous connection between its motivating problem and the proposed formulation. Specifically, the authors do not adequately justify modeling heterogeneous distributions as a mixture of ID, covariate shift, and semantic OOD components, nor do they clearly establish the limitations of existing distribution-level UQ approaches that motivate the proposed sample-level formulation. More importantly, the relationship between the distributional objective in Eq. (1) and the sample-level formulation in Eq. (2) remains unclear. The authors provide neither a formal characterization of this relationship nor sufficient empirical evidence demonstrating that the latter adequately addresses the former. Furthermore, since the proposed plug-in module itself relies on Gaussian approximations and aggregate statistics, the theoretical distinction between the proposed sample-level approach and existing distribution-based methods is not sufficiently established.

2. Insufficient Evidence of Novelty and Complementarity in the Dual-Signal Design

The proposed method relies heavily on established components, particularly class-conditional Mahalanobis distance and regularized posterior variance, without sufficiently justifying their combination or the calibration architecture in Eqs. (11)–(15). The theoretical results (Theorem 2 and Proposition 3) primarily characterize properties of the underlying statistics but do not establish the complementarity of the two signals or provide theoretical guarantees for their combined calibration. This concern is also reflected in the empirical results: the improvements obtained by combining the two signals are marginal in some settings (e.g., AUROC differences of approximately 0.001–0.002 in Figure 4a), while the absence of variability estimates across multiple random seeds makes it difficult to assess whether these improvements are consistent and meaningful. Consequently, the current theoretical analysis and experimental evidence do not convincingly establish the necessity or practical benefit of the proposed dual-signal design.

Questions for authors

What specific limitations of existing distribution-level UQ methods necessitate the proposed sample-level reformulation? More importantly, what theoretical relationship exists between the distributional objective in Eq. (1) and the sample-level formulation in Eq. (2)? Can the authors provide a formal justification or empirical evidence showing that the proposed reformulation adequately captures the original distributional heterogeneity?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.