- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper addresses calibration degradation of uncertainty quantification (UQ) under heterogeneous mixed distributions, where in‑distribution, covariate‑shifted and semantic OOD samples are mixed together. It proposes a lightweight plug‑in module D‑CUQ built on frozen pre‑trained model features. Two signals are computed: manifold‑deviation signal measures positional offset from ID manifold, and coverage‑gap signal evaluates local geometric coverage given training data. The paper builds theoretical connection between sample‑level signals and distribution‑level shift notions, and designs an entropy‑guaranteed calibaration strategy. It works with existing UQ baselines without changing classification labels or regression prediction mean. Experiments and ablations on classification and regression tasks verify the joint improvement brought by dual‑signal.
Critical strengths
- The ablation suite is carefully designed to exclude key alternative explanations. The shuffled‑dual control proves performance gain roots in sample‑wise paired dual‑signal, instead of mere output smoothing. Comparisons against single‑signal and global scaling further confirm both signals are necessary. These empirical evidences well support the validity of this plug‑in method.
- This work implements both classification and regression to verify the generality of feature‑based dual‑signal. It works as plugin on top of frozen backbone without retraining, compatible with multiple mainstream UQ baselines. For regression task, target binning adaptation makes dual‑signal mechanism function without relying on specific output head, which broadens its applicable scope.
Critical weaknesses
- The key solution is incremental combinatorial innovation, and its theoretical correspondence only holds for one‑way operational property. Underlying operators are from prior works (Lee et al.,2018; Mukhoti et al.,2023). The sample‑distribution relation has counter‑examples and cannot equal true sample shift generation type. The UQ degradation under mixed distribution is observed in Ovadia et al. (2019); full‑spectrum OOD concept from Yang et al. (2023), OpenOOD v1.5 benchmark from Zhang et al. (2024).
- Evaluations reply on self‑constructed heterogeneous mixed test set, and the method has relatively narrow applicable scenarios. There exists no community‑standard benchmark, raising barrier for fair cross‑method comparison. As post‑processing plug‑in, it cannot fix representation defects and heavily relies on feature quality from frozen model, only valid under constrained setup.
Questions for authors
- Ablation only proves signal utility. How to empirically verify signal correspondence with two types of shift generation mechanism?
- Why there is no quantitative experiment comparing against Geometric‑ODIN which also uses dual scores for shift characterization?
- Could you provide quantitative results of D‑CUQ when backbone representation quality degrades?
- Under your operational definition, what is approximate fraction of counter‑examples in real test?