acceptodds
Under review as a conference paper at ICLR 2027

Evaluating Multimodal Inverse Problems: A Protocol for Per-Event and Population-Level Fidelity

Abstract

Evaluation in inverse problems is dominated by pointwise metrics - RMSE, MAE, per-event resolution - under the implicit assumption that lower error means better reconstruction. That the conditional mean poorly represents multimodal structure is well known but existing evaluation protocols do not score its consequences for the reported marginal. When one inverse problem is instantiated over many independent events and the scientific deliverable is the marginal over the dataset, as in unfolding, a second failure mode appears that pointwise metrics cannot capture. By the law of total variance, point estimators trained to minimize MSE produce a marginal spectrum strictly narrower than the truth whenever the posterior has nonzero width. The resulting bias is independent of architecture, training, and dataset size, and it compresses precisely the spectral features like shapes, tails and modes that scientific measurements rely on. We propose a three-part evaluation protocol where each step targets a failure mode the others miss: per-event distributional accuracy via CRPS, population-level marginal accuracy via a spectrum-fidelity diagnostic, and uncertainty trustworthiness via coverage-based calibration. We instantiate the protocol on a synthetic benchmark with an analytic posterior to verify the mechanism, and on a realistic many-to-one inverse problem from particle physics we show that model rankings reverse between pointwise and distributional metrics, and calibration further separates architectures indistinguishable under CRPS. The choice of evaluation protocol can therefore determine the scientific conclusion.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
evaluation protocol, inverse problems, proper scoring rules, calibration, normalizing flows, scientific machine learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_2FLsOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose a three-pronged protocol for evaluating solutions to inverse problems. (1) The first metric is CRPS, which evaluates distributional accuracy of the posterior recovered for one event. (2) The second is spectrum fidelity, which evaluates distributional accuracy of the marginal distribution recovered across a population of events. (3) The third is coverage, which evaluates how trustworthy a model’s stated uncertainty is. The authors apply this framework on simulated data in two settings, one with an analytical bimodal posterior, and one with an unknown bimodal posterior motivated by an application in particle physics. They find that expressive flow-based models do better than point reconstructors or simpler variational families on per-event and population-level distributional accuracy.

Critical strengths

This paper addresses an important problem in the deep inverse problems literature, which is that many of them report the wrong metrics. The biggest benefit I can see with this paper is that it adds pressure to future work to spend more time evaluating posterior distributional accuracy and uncertainty calibration.

Critical weaknesses

There’s nothing novel in this paper. The three metrics are pretty standard: plenty of papers use CRPS, binned population-level statistics, and/or coverage tests, although not necessarily all three at once. Furthermore, it’s not at all surprising that rich flow-based models are superior to point reconstructors or simpler variational families like a mixture of Gaussians.

My main concern is about the soundness of relying on these three metrics solely. The paper doesn’t verify whether they are necessary and sufficient metrics. If not through a theoretical argument, it would be nice if the authors could at least verify the sufficiency of these metrics by showing correlation with other well-established distributional accuracy metrics like KL divergence, Wasserstein distance, maximum mean discrepancy (MMD), etc. Furthermore, this paper only addresses one-dimensional distributions, making it difficult to draw any conclusions about its applicability to more complex real-world inverse problems with high-dimensional data (e.g., imaging).

Questions for authors

None

1 comment
AuthorsOpenReview ↗

Response to Reviewer 2FLs

We thank the reviewer. The MDN matches the flow in Benchmark I (CRPS 1.286 vs 1.285) so our point is posterior expressivity. Appendix D.11 covers the full 8D latent, and Table 10 already reports the Wasserstein-1 (EMD) distance where the rankings are identical.

Kind regards, The authors

Reviewer_VQujOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper argues that pointwise estimators, such as RMSE, are poorly suited to solving ill-posed inverse problems with wide or multimodal posteriors. This is especially true when one problem is solved over many events and the scientific deliverable is the population-level distribution of the latent variable. By the law of total variance, any estimator converging to the conditional mean yields a reconstructed distribution systematically narrower than the truth, and this bias does not vanish with more data.

The authors propose a three-part protocol to expose such failures: per-event distributional accuracy via the CRPS, population-level fidelity via a binned chi-squared comparison of spectra, and uncertainty calibration via conformal coverage curves. On a synthetic problem with an analytic bimodal posterior and on simulated top-quark pair reconstruction, model rankings reverse. MSE regressors win on RMSE, while normalizing flows dominate the distributional and spectral metrics, and calibration further separates two flows that are otherwise indistinguishable.

Critical strengths
  1. The paper’s most valuable message is that the minimum mean squared error estimator, although optimal for the loss it minimizes, is the wrong choice whenever the scientific deliverable is a distribution or a nonlinear function of the latent variable. The authors show that posterior-preserving models give better answers to ill-posed inverse problems. A single posterior sample per event faithfully reproduces the population marginal, whereas the conditional mean cannot. The Jensen-gap analysis in the appendix further strengthens the point: once the estimator is formed in the space of the downstream observable rather than the latent space, averaging flow posterior samples beats direct MSE regression even in terms of RMSE. This result deserves more prominence in the main text.

  2. The second strength is the protocol itself, which is constructive and well motivated. Each of its three axes targets a distinct failure mode: per-event distributional accuracy, population-level spectral fidelity, and uncertainty calibration. The authors explicitly demonstrate that the axes are not redundant, with models matched on CRPS that differ sharply in spectral fidelity and models that are equally well calibrated but differ greatly in sharpness. The protocol applies across model families, since the CRPS reduces to the mean absolute error for point estimators. The validation strategy is sound, moving from a synthetic problem with an analytic posterior to a realistic particle-physics problem, with robustness checks on binning, ensemble size, and sampling conventions, and with code and precomputed results released for reproducibility.

Critical weaknesses

The main limitation is scalability to high-dimensional latent spaces, such as imaging problems with millions of unknowns. The per-event score is essentially univariate. Its multivariate extension, the energy score, is tested only in eight dimensions on a single seed. It is known to be weakly sensitive to errors in the correlation structure as the dimension grows, and it depends on an arbitrary normalization of the components. The spectrum-fidelity axis relies on histograms and therefore cannot be applied to more than a few dimensions. The authors acknowledge that unbinned joint alternatives have not been validated, leaving open the question of how to choose summary observables without cherry-picking. Conditional calibration in high dimensions is likewise not addressed.

Compounding this, the computational cost of the protocol is not discussed in the main text. Drawing hundreds of posterior samples per event over tens of thousands of test events already requires about thirty GPU-hours for this low-dimensional problem, a figure that appears only in the reproducibility appendix. For diffusion-based reconstruction in medical imaging, evaluation can cost far more than training, and the paper offers no cost model or guidance on the minimum number of samples needed to preserve rankings.

Several further weaknesses temper the conclusions. The theoretical argument is a direct consequence of the law of total variance, so the novelty lies mainly in the protocol and its demonstration rather than the theory. The baselines are limited. The synthetic MSE regressor trivially predicts zero by symmetry, and there are no mode-seeking estimators, quantile regression, ensembles, diffusion samplers, or MCMC reference posteriors. The calibration results raise a puzzle: split conformal prediction should guarantee marginal coverage, yet both flows systematically undercover in the physics benchmark. Moreover, the calibration figures there come from a single seed. The chi-squared statistic is acknowledged to be biased for heavy-tailed observables. Finally, the claimed impact on scientific measurements is argued but never demonstrated by propagating reconstructions into an actual parameter fit. The physics benchmark also uses only simulated data, so robustness to real collision data remains untested.

Questions for authors

1.Why were MCMC samplers not considered as baselines or reference posteriors? In the synthetic benchmark, the analytic posterior is available, so scoring it directly or sampling it via MCMC would yield Bayes-optimal values for each metric. In the physics benchmark, where the likelihood is intractable, could likelihood-free MCMC or MCMC under an approximate transfer-function likelihood, as in matrix-element methods, serve as a principled reference?

  1. Given that InverseBench is cited, why were diffusion-based posterior samplers, deep ensembles, or quantile regression not included among the compared methods? Would the observed reversal in rankings hold for these families?

  2. Could the authors report the total computational cost of the protocol relative to training? How does this cost scale with the dimension of the latent space, the number of posterior samples per event, and the size of the test set? What is the smallest number of samples that preserves model rankings for each metric?

  3. How would the protocol be instantiated for latent spaces with millions of variables, as in medical image reconstruction? In particular, how should summary observables be chosen for the spectrum-fidelity axis, which unbinned distance would be appropriate, and does the energy score retain its discriminative power as dimension grows?

  4. Split conformal prediction guarantees marginal coverage under exchangeability, yet both flows systematically undercover in the physics benchmark. Are the reported coverage curves actually conformalized, or are they based on raw likelihood regions? If conformalized, what explains the violation?

  5. Could the calibration results of the physics benchmark be reported over several seeds, given that the distinction between the two flows rests on them? Could the spectral comparison for heavy-tailed observables use the Baker–Cousins statistic rather than the Poisson-approximated chi-squared?

  6. Could the authors demonstrate that the differences in spectral fidelity translate into a measurable bias in an actual physics parameter, for example, a spin-correlation fit based on the azimuthal-angle observable?

3 comments
AuthorsOpenReview ↗

Response to Reviewer VQuj

We thank you for the careful review. We address your questions one by one:

Q1. In Benchmark I, where the posterior is known analytically, we will add an oracle that shows the best achievable value of each metric. For Benchmark II, we agree that a matrix-element method would be the principled reference but this is beyond the scope of this work.

Q2. We will add a deep ensemble and quantile regression to Benchmark I, and the deep ensemble also to Benchmark II, where quantile regression cannot represent the joint latent.

Q3. The sampling took about 30 GPU-hours (Appendix E). The CRPS rankings are stable from about 100 samples (Fig. 6).

Q4. We agree. Sections 1 and 7 and Appendix D.11 already limit our scope to one-dimensional scored quantities. In imaging, the deliverable is usually a single reconstruction, where the protocol reduces to Axes I and III (Section 7). Axis II applies when a population of derived quantities is the deliverable, as in high energy physics, where per-event reconstructions are aggregated into cross sections, or for galaxy redshift distributions from aggregated photometric redshifts. The choice of unbinned joint distance, and the energy score's weaker sensitivity in high dimensions, remain open questions.

Q5. The curves are conformalized, but there is an error in our description. Following Araz & Spannowsky (2025), we calibrate on true events and test flow samples against that threshold (conformal outlier test). If the flow matches the data, coverage is nominal, so deviations measure model fidelity. We made the mistake of calling this coverage and reading undercoverage as too narrow. It means flow samples are less typical than real events: posteriors that are too wide, or a mismatch between sampler and density. We will correct the description in Section 4.3 and Appendix D.10.

Q6. Yes to both.

Q7. We agree this would be the best demonstration and leave the folding test of arXiv:2611.09797 to future work. In the revision, we will rephrase the downstream claim in Section 3.2 as reduced sensitivity rather than bias.

Kind regards, The authors

Reviewer_VQujOpenReview ↗

reply to reviewers' comments

Thanks to the authors for their compelling, detailed, and honest responses. Adding an oracle (Q1) and a DL/quantile regression to Benchmark I are very compelling aspects from my perspective.

Follow-up on Q4: Your statement in wrong in multiple aspects. In imaging there are many situations where you're NOT interested in a single reconstruction; for instance where you're doing quantitative MRI, you need to reconstruct multi-echo images from which you want to extract parametric maps (for multiple parameters), similarly, in functional MRI or diffusion-weighted MRI where the 4th dimension is time or angular resolution. On top of that, you also want to reconstruct uncertainty maps on top of the pointwise estimators. Last but not least, in PET imaging, you might be interested in reconstructing a 3D radiotracer probability density from which you can sample (see works on non-parametric Bayesian estimation in PET imaging).

Thank you for these examples. We see your point and agree. As far as we can tell, Axis II has a natural role in imaging wherever the distribution of voxel-wise parameters across a region is the deliverable, as in histogram analysis of quantitative and diffusion MRI maps. Non-parametric Bayesian PET appears to be an even closer match to our setting. The open problem becomes choosing and assessing the distance between distributions when voxels are spatially correlated or parameters are estimated jointly. We will note this in Section 7.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response to all reviewers

We thank all the reviewers. As the paper also states, the law of total variance is textbook. Our contribution is an evaluation protocol for per-event reconstructors whose population marginal is the deliverable (Table 1). On a realistic problem, the pointwise winner, the MSE + MMD transformer following CMS (2025a), loses under distributional metrics, also in the full 8D latent space (Appendix D.11).

In a revised version, we will add an analytic-posterior oracle, deep-ensemble and quantile-regression baselines as suggested by one reviewer. We will use the Baker-Cousins statistic for the heavy-tailed observables reported in the appendix (Table 12), and we will show the calibration results over all seeds (Table 3).

AuthorsOpenReview ↗

Response to Reviewer JhA5

We thank the reviewer. The theoretical identity is itself standard as we state in the paper. Our contribution is the protocol. The baseline follows a recent CMS approach. Additionally, a recent CMS Open Data study (arXiv:2611.09797) evaluates its flow only by the sample mean. The ranking flip holds in 8D (Appendix D.11) and our main observable in Benchmark II, Δϕ\Delta \phi, is not heavy-tailed.

Kind regards, The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.