When the Prior Is Wrong: Probing Misspecified Diffusion Priors in Inverse Problems

While diffusion priors generate high-quality posterior samples across many scientific inverse problems, they are often trained on purely simulated data, thus inheriting the errors and biases of these underlying sources. Many approaches to adapting diffusion priors from a single observation require sufficient conflict between the data and the prior to successfully adapt and do not allow specific misspecification hypotheses to be tested. On the other hand, existing out-of-distribution (OOD) detection methods determine whether the observation is compatible with a fixed prior, rather than specifying which assumptions to relax or by how much. To address these issues, we propose PIVOT: Prior Inference Via Observation-driven Transformations, which formulates diffusion prior misspecification as inference over an underlying latent variable representing a scientifically plausible discrepancy. By expanding the prior along these plausible directions of misspecification, PIVOT infers a posterior over the strength of each proposed discrepancy given the observation. This process can be efficiently amortized with just one posterior sampling pass using a conditional diffusion model. We also demonstrate how to determine if a proposed discrepancy is inconclusive because the measurement is too noisy or ill-posed. We validate PIVOT on seismic imaging and real-world black hole imaging problems, where we find that OOD observations generally lead to larger perturbation strengths when the proposed discrepancy family captures relevant components of the mismatch, as long as the discrepancy is visible to the measurement. PIVOT thereby provides an interpretable probe into how real-world priors could be misspecified.