- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper describes the identifiability problem in deep evidential regression. The main problem is that different evidential parameters can produce the same predictive distribution, while their estimated epistemic uncertainties are different. The authors looking for conditions this problem can be solved by using some restrictions on the model architecture. They illustrate theoretical results show that the parameters can be identified under specific conditions, by using fixed feature representations. They extend their analysis and experiments to multivariate regression by introducing a gauge-fixing approach. and do experiments on synthetic data, cifar rotation, uci datasets, and QM9.
Critical strengths
-
One important strength of this paper is that it provides a theoretical analysis of the problem in DER. The authors present necessary and sufficient conditions for identifying evidential parameters and also extend their results to multivariate regression. I think this is an important contribution because identifiability is a fundamental problem in the uncertainty estimation, and understanding this problem can help to develop more reliable evidential models.
-
Another strength of the paper is that the authors evaluate their approach on different synthetic and real-world datasets. The results show that the proposed method can generate more stable evidence and epistemic uncertainty estimates among different random seeds, while create comparable predictive performance in several settings. These experiments are really useful because they show that the theoretical findings can also have practical benefits, especially for improving of uncertainty estimates.
Critical weaknesses
1.One of the important weakness of the paper is that the theoretical results depend on some assumptions, and using fixed feature representations and specific architectures. Actually in practical deep learning models, the feature representations will be usually learned with the model parameters. The authors also mention that their theoretical guarantees do not directly apply to this setting. I think this limitation is important because it is related to the generalization of the mode, and also it is not clear how much the proposed results can be generalized to more architectures.
- Another weakness is related to the experimental results which mainly focus on the stability of evidential parameters on different random seeds, instead of showing that the estimated epistemic uncertainty is more accurate or reliable. However the proposed method improves stability in experiments, the synthetic results also show that stable parameters can be far from their true values. Also the predictive performance is not better than the baseline methods. I think this is an important concern because only stability improvement does not necessarily mean the model creates better uncertainty estimation in practical applications.
Questions for authors
-
The theoretical results mainly rely on fixed feature representations and specific assumptions on the feature distribution. However, in practical deep learning applications, features are usually learned with the evidential parameters joint. Can the authors explain about the restriction of these assumptions in practice?
-
The experimental results show that the proposed method improves the stability of evidential parameters between different random seeds. However, it is not clear if this improvement also makes more accurate or uncertainty estimates. Can the authors provide more explanation about the relationship between parameter stability and the quality of uncertainty estimation?
2 comments
Dear Reviewer oiow, with about a day left in the discussion, we would like to point briefly to the two results that answer your questions directly (revised PDF, Appendix L).
- Learned features. Freezing the trunk of a standard jointly trained DER at its best epoch and refitting our head places it in the fixed-feature setting. The refit head forgets its initial evidence (memory b ≈ 0), while the matched scale-closed head retains about a third of it, and the refit head improves test NLL over the source DER (1.94–1.95 vs 2.00–2.05).
- Stability vs. accuracy. Against known ground truth, the identifiable head recovers the true epistemic variance (relative error 0.007–0.069), while a scale-closed head with the same test NLL (|Δ| < 10⁻⁵) does not (error up to 1.60).
If these resolve your concerns, we would be grateful if you would consider updating your assessment. If anything remains unclear, we would value hearing it so we can respond before the deadline.
Dear Reviewer oiow,
Thank you again for your review. With the discussion period closing on November 18, we would like to check whether our response addressed your two questions. In short:
Details are in Appendix L of the revised PDF. We would be happy to clarify anything before the deadline, and if these results address your concerns, we would appreciate it if you would consider updating your score.