acceptodds
Under review as a conference paper at ICLR 2027

Identifiability in Deep Evidential Regression: Scalar Boundaries and a Multivariate Gauge

Abstract

Deep Evidential Regression (DER) gives single-pass uncertainty estimates, but different evidential parameters can induce exactly the same Student- predictive distribution while reporting different evidence and epistemic uncertainty. We characterize when architectural restrictions remove this ambiguity. If the evidence strength is shared across inputs, the parameters are identifiable exactly when no two distinct allowed scale functions are positive constant multiples of one another. If both and are represented by affine softplus heads on fixed features, they are jointly identifiable under an open-set density condition unless both heads are constant. For multivariate DER, we decompose the prior scale matrix into a normalized shape, which the predictive law identifies, and a scalar amplitude governed by the same scalar criterion. Experiments distinguish this structural property from finite-sample behavior. The corresponding heads yield more reproducible evidence strengths on CIFAR-rotation and UCI regression tasks, with predictive quality evaluated alongside stability. In a joint-feature QM9 experiment, the gauge-fixed parameterization reduces variation in the epistemic covariance trace relative to coupled and uncoupled Normal-Inverse-Wishart baselines at the final checkpoint. Together, these results show how architectural structure can make evidential parameters uniquely determined and more reproducible in practice.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
deep evidential regression, identifiability, uncertainty quantification, predictive ambiguity, gauge fixing

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_oiowOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper describes the identifiability problem in deep evidential regression. The main problem is that different evidential parameters can produce the same predictive distribution, while their estimated epistemic uncertainties are different. The authors looking for conditions this problem can be solved by using some restrictions on the model architecture. They illustrate theoretical results show that the parameters can be identified under specific conditions, by using fixed feature representations. They extend their analysis and experiments to multivariate regression by introducing a gauge-fixing approach. and do experiments on synthetic data, cifar rotation, uci datasets, and QM9.

Critical strengths
  1. One important strength of this paper is that it provides a theoretical analysis of the problem in DER. The authors present necessary and sufficient conditions for identifying evidential parameters and also extend their results to multivariate regression. I think this is an important contribution because identifiability is a fundamental problem in the uncertainty estimation, and understanding this problem can help to develop more reliable evidential models.

  2. Another strength of the paper is that the authors evaluate their approach on different synthetic and real-world datasets. The results show that the proposed method can generate more stable evidence and epistemic uncertainty estimates among different random seeds, while create comparable predictive performance in several settings. These experiments are really useful because they show that the theoretical findings can also have practical benefits, especially for improving of uncertainty estimates.

Critical weaknesses

1.One of the important weakness of the paper is that the theoretical results depend on some assumptions, and using fixed feature representations and specific architectures. Actually in practical deep learning models, the feature representations will be usually learned with the model parameters. The authors also mention that their theoretical guarantees do not directly apply to this setting. I think this limitation is important because it is related to the generalization of the mode, and also it is not clear how much the proposed results can be generalized to more architectures.

  1. Another weakness is related to the experimental results which mainly focus on the stability of evidential parameters on different random seeds, instead of showing that the estimated epistemic uncertainty is more accurate or reliable. However the proposed method improves stability in experiments, the synthetic results also show that stable parameters can be far from their true values. Also the predictive performance is not better than the baseline methods. I think this is an important concern because only stability improvement does not necessarily mean the model creates better uncertainty estimation in practical applications.
Questions for authors
  1. The theoretical results mainly rely on fixed feature representations and specific assumptions on the feature distribution. However, in practical deep learning applications, features are usually learned with the evidential parameters joint. Can the authors explain about the restriction of these assumptions in practice?

  2. The experimental results show that the proposed method improves the stability of evidential parameters between different random seeds. However, it is not clear if this improvement also makes more accurate or uncertainty estimates. Can the authors provide more explanation about the relationship between parameter stability and the quality of uncertainty estimation?

2 comments

Dear Reviewer oiow,

Thank you again for your review. With the discussion period closing on November 18, we would like to check whether our response addressed your two questions. In short:

  1. Learned features: freezing the trunk of a standard, jointly trained DER at its best epoch and refitting the head satisfies the fixed-feature setting. The refit head then forgets its initial evidence (memory b = -0.06 to 0.00, vs 0.34-0.37 for a scale-closed control) and improves test NLL over the source DER (1.94-1.95 vs 2.00-2.05).
  2. Stability vs. uncertainty quality: on the known-truth generator, our head recovers the true evidence (0.50 / 1.00 / 2.01 / 5.03 for ν* = 0.5 / 1 / 2 / 5) with epistemic relative error 0.007-0.069, while a scale-closed head with the same test NLL (|Δ| < 1e-5) has errors up to 1.60.

Details are in Appendix L of the revised PDF. We would be happy to clarify anything before the deadline, and if these results address your concerns, we would appreciate it if you would consider updating your score.

Dear Reviewer oiow, with about a day left in the discussion, we would like to point briefly to the two results that answer your questions directly (revised PDF, Appendix L).

  1. Learned features. Freezing the trunk of a standard jointly trained DER at its best epoch and refitting our head places it in the fixed-feature setting. The refit head forgets its initial evidence (memory b ≈ 0), while the matched scale-closed head retains about a third of it, and the refit head improves test NLL over the source DER (1.94–1.95 vs 2.00–2.05).
  2. Stability vs. accuracy. Against known ground truth, the identifiable head recovers the true epistemic variance (relative error 0.007–0.069), while a scale-closed head with the same test NLL (|Δ| < 10⁻⁵) does not (error up to 1.60).

If these resolve your concerns, we would be grateful if you would consider updating your assessment. If anything remains unclear, we would value hearing it so we can respond before the deadline.

Reviewer_VwCcOpenReview ↗
Rating
3
Confidence
2(−1)
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies predictive identifiability in Deep Evidential Regression: when does the Student-t predictive distribution uniquely determine its underlying evidential parameters? For global evidence strength, Theorems 3.1–3.2 characterize identifiability through scale-separation of the admissible scale-function class. Theorem 3.3 establishes identifiability of affine-softplus head pairs that are not both constant, on fixed features satisfying an open-set density condition. Theorems 3.4–3.5 extend the analysis to multivariate regression by separating the normalized shape of the NIW scale matrix from the remaining scalar ambiguity. Synthetic experiments examine parameter recovery, while CIFAR-rotation, UCI and QM9 experiments compare cross-seed stability alongside predictive quality.

Critical strengths
  1. The architectural identifiability result provides useful new knowledge. Theorem 3.3 addresses a substantive question left open by analyses of unrestricted evidential outputs: shared head structure can exclude otherwise predictively equivalent alternatives. Lemma B.3 supplies an explicit factorization argument, including the different slope-sign cases and the constant-head exception (Appendix B, pp. 13–16). The global characterization and multivariate reduction organize the result into a coherent account of where ambiguity persists and where it disappears. This precise structural contribution is the principal reason to support acceptance.

  2. The interpretation is appropriately disciplined. The paper distinguishes predictive injectivity, optimization stability, parameter recovery and epistemic faithfulness. It reports strongly biased but concentrated synthetic estimates, the coupled NIW baseline's superior evidence-amplitude stability on QM9, and dataset-dependent predictive tradeoffs (Sections 4.2–4.4; Appendices D.1–D.3; Tables 14–18). These distinctions prevent the empirical stability results from being presented as proof of accurate or Bayesian uncertainty. The negative findings help clarify the significance and limits of the theoretical result.

Critical weaknesses
  1. The stability comparisons do not fully isolate the effect of removing predictive ambiguity. Most comparisons replace a per-input evidence head with a global scalar and initialize evidence at one (Section 4; Appendix J.1; Table 19). The two-stage UCI head discussed in the main text additionally differs from the vanilla baselines in its feature-training protocol; the jointly trained Arm B in Table 15 controls for this and shows similarly low dispersion, so the UCI result does not depend on that protocol, but the global-versus-per-input and common-initialization differences remain in every real-data comparison. Reduced flexibility and a common starting point could contribute to lower cross-seed dispersion. The scale-closed diagnostics (Appendices D.4 and D.6) are the natural control, but they are run only under a diverse-initialization protocol with different budgets, and the multi-initialization check in Appendix D.1 fixes γ and α to their generating values; neither establishes that orbit removal explains the main method comparison. A scale-closed counterpart of the global head trained under the common-initialization protocol with the same budget, and the scale-separated head under the diverse-initialization protocol with learned γ and α, would substantially strengthen the practical interpretation.

  2. The real-data evidence has a limited connection to the theorem's applicability and to useful uncertainty estimation. The CIFAR and UCI feature supports do not satisfy the stated density condition, QM9 jointly learns its representation (Appendices D.7, F and H), and all real-data runs use the global-evidence head of Theorems 3.2 and 3.4, so the per-sample result of Theorem 3.3 is evaluated only on synthetic data (Section 4.4; Table 19). Moreover, CIFAR-rotation prediction improves only slightly over an image-free reference (Table 9), while synthetic recovery can remain biased despite longer optimization (Appendix D.1). These limitations are explicitly acknowledged and do not invalidate the stated theorem, but they leave the practical value less established than the structural result. They support weak rather than clear acceptance.

Questions for authors
  1. Which reported comparison most directly isolates scale-separation from the use of a global evidence head and from common evidence initialization? If no matched comparison is available, please confirm that the CIFAR-rotation and UCI stability claims, like the QM9 claim in Appendix H, should be read as comparisons of complete parameterizations rather than as evidence that orbit removal is the mechanism.
  2. Which existing result most directly demonstrates useful conditional uncertainty beyond improved cross-seed reproducibility, given Table 9 and the finite-sample recovery behavior in Appendix D.1?
4 comments

Dear Reviewer VwCc,

Your review identified precisely the experiment that was missing, and it has made the paper's central claim considerably stronger.We are grateful for this.

We ran both controls as described: a scale-closed counterpart of the global head with the same initialization, data order and budget, and both heads trained from five initial evidence levels with γ and α learned. On CIFAR-rotation our head forgets its start (memory b = -0.02 to 0.08), while scale-closed, which is one scalar more flexible, retains about 40% of it (b = 0.38-0.43) at a similar predictive fit. On UCI our head forgets faster on all nine datasets. Following your point, the revised paper presents the common-initialization tables as comparisons of complete parameterizations and rests the mechanism claim on this matched experiment (Appendix L, revised §4).

As the discussion period ends on November 18, we would be grateful to know whether this resolves your concern, and we are glad to answer any follow-up questions.

Dear Reviewer VwCc, thank you again for specifying the two controls. We ran both as described (Appendix L): (a) a scale-closed counterpart under the common-initialization protocol with the same budget, and (b) our head under diverse initialization with γ and α learned. Under common initialization the two heads are equally reproducible, so that comparison cannot discriminate them, as you anticipated. Under diverse initialization our head forgets its start (b = 0.01 / 0.08 / −0.02), while the scale-closed head, one scalar more flexible, keeps about 40% (b ≈ 0.4) at similar predictive fit. This attributes the difference to scale-separation rather than reduced flexibility. We would appreciate a brief note on whether this addresses your concern, or what gap you still see.

We thank the authors for their response and additional experiments. We believe some issues have been resolved, and we keep the weak acceptance score.

Thank you for the acknowledgement. Your review tied the step from weak to clear acceptance to two points, and we want to make sure our answers reached them.

  1. Isolating the mechanism. We ran both controls you specified. In the matched pair (same global head, initial weights, data order and budget; the control adds one learnable multiplier on β, which makes the class scale-closed), our head forgets its initial evidence (memory b = 0.01 / 0.08 / −0.02 over three data realizations), while the control, one scalar more flexible, keeps about 40% of it (0.41 / 0.43 / 0.38). As you asked, the submitted common-initialization tables are now described as comparisons of complete parameterizations, and the mechanism claim rests on this matched experiment (Appendix L and a new main-text figure).

  2. Real data and useful uncertainty. The Theorem 3.3 class now has real-data results on CIFAR features and QM9 embeddings, and every fitted real-data head leaves a nonzero residual when the class tries to absorb a factor-2 rescaling of the evidence (0.04–0.37; 0 would mean absorbed). On data with known truth, heads with the same test NLL (|Δ| < 10⁻⁵) report epistemic variances set by their initialization unless the class is identifiable: trained to convergence, ours recovers the true epistemic variance (relative error 0.007–0.069), while the scale-closed control errs by up to 1.60.

If a concern behind your assessment is still open, could you tell us which one? We can answer from the experiments already run while the discussion is open.

Reviewer_f5HGOpenReview ↗
Rating
3
Confidence
1
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies when Deep Evidential Regression parameters are uniquely determined by the predictive distribution. It establishes a necessary and sufficient scale-separation condition for global evidence strength and proves identifiability for nonconstant softplus-affine heads under fixed-feature density assumptions. It extends these results to multivariate regression by separating covariance shape from scale. Experiments show improved reproducibility of evidence and epistemic uncertainty estimates, while emphasizing that identifiability does not guarantee accurate parameter recovery or faithful epistemic uncertainty.

Critical strengths
  1. Under a global scalar evidence strength, the paper establishes scale-separation of the β-function class with respect to the input marginal P_X as a necessary and sufficient condition for injectivity of the predictive map and hence for identifiability of the evidence strength. Under fixed features and the stated density condition, it also precisely identifies the only exception for per-sample softplus-affine heads: when both heads are constant, only the predictive combination βc(ν) is identified, leaving a one-parameter ambiguity.
  2. A degree-1 positive-homogeneous gauge decomposes the NIW scale matrix into a normalized shape and a scalar amplitude, elegantly reducing multivariate identifiability to the scalar problem without requiring a fixed ratio between evidence parameters.
  3. The experiments cover synthetic parameter recovery and several real-world benchmarks, and the paper explicitly distinguishes improved reproducibility from accurate parameter recovery and faithful epistemic uncertainty. This keeps the conclusions appropriately scoped and avoids equating identifiability with Bayesian or epistemic faithfulness.
Critical weaknesses

w1. The identifiability results for the proposed softplus heads require fixed features and a feature distribution satisfying the stated density condition. These assumptions may be difficult to satisfy with learned representations, particularly when features lie on low-dimensional manifolds. The theory also leaves joint feature learning unresolved.

w2. The gap between identifiability and practical parameter recovery remains substantial. When the true global evidence strength is ν* = 5, the mean estimate is approximately 1.18 under the original training budget. The additional recovery diagnostics help explain this gap, but do not establish when accurate recovery can be expected with finite data and computation.

w3. Replacing per-sample evidence with a global scalar reduces model flexibility, which may itself improve stability across seeds. The UCI experiments include both two-stage and jointly trained variants, but the comparisons still do not fully separate the benefits of identifiability from those of restricting evidence to a single shared value.

w4. The main stability comparisons initialize all models with the same evidence strength, so their conclusions may depend on this choice. Multiple initializations are examined only in a separate known-nuisance diagnostic, leaving robustness of the complete models unclear. The per-sample identifiability result also receives limited validation on real data.

Questions for authors
  1. The authors report that at ν∗=5\nu^*=5 the learned global head estimates around 1.181.18, whereas recovery improves substantially when the nuisance parameters are fixed to their true values, and Appendix D.1 reports longer-budget results. Could the authors clarify what these diagnostics jointly suggest about the main source of the recovery gap? In particular, which part should be attributed to optimization, finite-sample estimation, or learning the nuisance heads, and which part remains unresolved?

  2. The authors acknowledge that replacing per-sample evidence with a global scalar reduces model flexibility, and they point to the UCI training variants and the scale-closed diagnostic in Appendix D.6. Could the authors clarify how far the existing results actually support attributing the stability gains specifically to identifiability, rather than to the potentially stabilizing effect of restricting evidence to a global scalar? What do these comparisons establish beyond that restriction?

  3. The main stability comparisons use a common initial evidence strength, while the initialization diagnostics examine different and more restricted settings. Could the authors clarify how broadly readers should interpret the reported stability advantage? Specifically, does sensitivity to initial evidence remain an open question for the complete models, and how does that limit the practical conclusions?

2 comments

Dear Reviewer f5HG,

Thank you again for your review. As the discussion period ends on November 18, we wanted to check briefly whether our responses answered your questions. In particular, the 1.18 gap is mainly unfinished optimization: at a stationary point our head recovers ν* = 5 as 5.03 ± 0.17 at n = 128k, independent of its initialization. A matched scale-closed control further shows that the stability comes from scale-separation rather than from the global restriction alone. We are happy to clarify anything further.

Dear Reviewer f5HG, a short pointer before the discussion closes: the ν* = 5 gap comes from unfinished optimization, since the paper's 1000-epoch budget leaves the evidence near its starting value. At a stationary point the head recovers 5.03 ± 0.17 at n = 128k, and the matched scale-closed control in Appendix L separates scale-separation from the global restriction. We would be glad to clarify anything further.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.