- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors propose a three-pronged protocol for evaluating solutions to inverse problems. (1) The first metric is CRPS, which evaluates distributional accuracy of the posterior recovered for one event. (2) The second is spectrum fidelity, which evaluates distributional accuracy of the marginal distribution recovered across a population of events. (3) The third is coverage, which evaluates how trustworthy a model’s stated uncertainty is. The authors apply this framework on simulated data in two settings, one with an analytical bimodal posterior, and one with an unknown bimodal posterior motivated by an application in particle physics. They find that expressive flow-based models do better than point reconstructors or simpler variational families on per-event and population-level distributional accuracy.
Critical strengths
This paper addresses an important problem in the deep inverse problems literature, which is that many of them report the wrong metrics. The biggest benefit I can see with this paper is that it adds pressure to future work to spend more time evaluating posterior distributional accuracy and uncertainty calibration.
Critical weaknesses
There’s nothing novel in this paper. The three metrics are pretty standard: plenty of papers use CRPS, binned population-level statistics, and/or coverage tests, although not necessarily all three at once. Furthermore, it’s not at all surprising that rich flow-based models are superior to point reconstructors or simpler variational families like a mixture of Gaussians.
My main concern is about the soundness of relying on these three metrics solely. The paper doesn’t verify whether they are necessary and sufficient metrics. If not through a theoretical argument, it would be nice if the authors could at least verify the sufficiency of these metrics by showing correlation with other well-established distributional accuracy metrics like KL divergence, Wasserstein distance, maximum mean discrepancy (MMD), etc. Furthermore, this paper only addresses one-dimensional distributions, making it difficult to draw any conclusions about its applicability to more complex real-world inverse problems with high-dimensional data (e.g., imaging).
Questions for authors
None
Response to Reviewer 2FLs
We thank the reviewer. The MDN matches the flow in Benchmark I (CRPS 1.286 vs 1.285) so our point is posterior expressivity. Appendix D.11 covers the full 8D latent, and Table 10 already reports the Wasserstein-1 (EMD) distance where the rankings are identical.
Kind regards, The authors