acceptodds
Under review as a conference paper at ICLR 2027

Livin' on a Prior: Likelihood Score Approximation for Inverse Problems

Abstract

Generative models have found great success as data-driven methods of solving inverse problems. Two popular approaches work either by combining a pretrained generative prior with a known degradation model, or by training a conditional generative model directly from paired data. We target a setting that spans both regimes: unknown degradations can be learned from few paired examples, while known degradations can be learned from self-generated samples. We introduce Likelihood Score Approximation (LSA), a generative framework that keeps a pretrained unconditional model fixed and learns an observation-conditioned model that approximates the likelihood score from paired samples. Within a conditional stochastic-interpolant framework, LSA can be trained in either score or velocity coordinates, independently of the unconditional model's native parameterization, and supports both deterministic and stochastic sampling. We further show empirically that the prior model can be swapped post-training while keeping the same LSA model. Across speech and image inverse problems, LSA operates effectively even at roughly 0.01% of the full training dataset. On the ImageNet-256 benchmark it achieves competitive or better restoration quality than strong posterior-sampling baselines while requiring up to several orders of magnitude fewer network evaluations.

Primary area:
generative models
Keywords:
inverse problems, diffusion models, flow matching, score-based generative models, posterior sampling, stochastic interpolants, data-efficient generative modeling

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_GUJbOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper presents Likelihood Score Approximation (LSA), a method for using diffusion models as priors in two common settings: (1) when the degradation operator is known, and (2) when the degradation is unknown but paired data are available. The authors build on the decomposition of the posterior score into the prior score and the gradient of the log-likelihood at each diffusion step. Since the prior score is readily available from a pretrained diffusion model, they propose amortizing likelihood score estimation by training a network on paired data to predict it from the current diffusion state and the degraded measurement. When the degradation operator is known, paired data are generated using the prior model. The paper evaluates LSA against a range of inverse-problem and restoration baselines across multiple speech and image tasks.

Critical strengths
  • Method: The paper unifies two common inverse-problem settings within a single framework. To my knowledge, using pretrained diffusion models for image restoration with unknown degradation operators and limited paired data is novel and addresses an important limitation of existing methods. The proposed approach handles both settings elegantly and is conceptually simple, which I consider a strength.
  • Clarity: The paper is clearly written and easy to follow. Most relevant information is provided in the main text or appendix.
  • Experiments: The experimental evaluation is comprehensive, covering a variety of inverse-problem tasks on both speech and image data.
Critical weaknesses
  • Evaluation: My main concern is the evaluation protocol.
    • I do not believe that 100100 samples are sufficient to reliably demonstrate improvements on the reported metrics, particularly FID. Although this protocol follows DAPS, I consider it inadequate and do not think it should be retained. I would recommend evaluating on at least 10241024 samples.
    • The baseline comparisons are too sparse to establish the method’s strengths convincingly. I would prefer consistent comparisons across all tasks against three strong, recent baselines for each data modality and inverse-problem setting (i.e, known and unknown degradation operators). Although the comparison with ControlNet is important, I believe it is better treated as an ablation than as a separate baseline. Furthermore, the advantages of LSA over RED-diff and DMAP+CSE, which share similarities with the proposed approach, are not convincingly demonstrated by a single comparison with each method. I am also concerned that applying restart-based refinement only to LSA gives it an unfair advantage, since this component is orthogonal to the proposed method.
    • The appendix reports an extensive hyperparameter search (p. 20, line 1029), raising concerns about the method’s sensitivity to hyperparameter choices and the effort required to identify a suitable configuration. It is also unclear which hyperparameters were tuned for the baselines and which configurations were ultimately used.
  • Related work: The paper omits a substantial body of related work on inverse problems and image restoration. In addition, DiffPhase, StreamFlow, and AnyEnhance are included in the comparisons without corresponding citations. See, for instance, the related work in [1] which should also be included as a reference.
  • Method: Although I appreciate the proposed approach, a minor limitation is its reduced generality. A desirable property of inverse problem methods, such as DPS, is that they can be applied across tasks without task-specific training or validation-based model selection.

[1] Idan Achituve, Hai Victor Habi, Amir Rosenfeld, Arnon Netzer, Idit Diamant, Ethan Fetaya Proceedings of the 42nd International Conference on Machine Learning, PMLR 267:420-443, 2025.

Questions for authors

Please refer to the Weaknesses section.

4 comments

We thank the reviewer for the critical assessment of our work. We appreciate the reviewer's recognition of the method's simplicity, novelty in the low-data setting, and broad applicability. We understand the concerns regarding the experimental evaluation and have made a substantial effort to address them. The feedback was particularly valuable, as the additional experiments and analyses motivated by your comments have significantly strengthened the paper. We hope the revised results provide a more convincing and transparent assessment of LSA. We encourage the reviewer to consult the revised manuscript for further details.

W1. Larger evaluation set

As requested, we extended all main ImageNet-256 results and all additional evaluations to 1,000 test images, including the updated Table 2 and Figure 5. The original DAPS benchmark evaluates only the first 100 images due to the substantial computational cost of posterior-sampling methods. Fortunately, the authors also provide an extended set of exactly 1,000 images, which we used with the same degradation protocols.

This set closely matches the size recommended by the reviewer and is publicly available, ensuring that our expanded evaluation remains directly reproducible.

The original 100-image evaluation results is used only for the comparison with ControlNet and RePS (Appendix D, Table 10), whose publicly available implementation was not accessible to us for a larger-scale evaluation, and also for a few ablations within the paper.

The ImageNet-512 and speech evaluations already used 1,000 test images and 886 long audio files respectively.

W2. Baselines and restart refinement

We appreciate the request for broader baseline coverage and have made a substantial effort to extend the comparisons across tasks and training regimes.

Known-operator image restoration.

We expanded the ImageNet-256 comparison to six baselines across all seven tasks (Table 2), including DAPS, RED-diff, DMAP, and several recent 2026 methods (DDiff, CLAMP, and PC-DAPS). All methods were evaluated on the same 1,000-image benchmark using consistent degradation operators and measurement noise. Notably, LSA still holds leading results in perceptual quality metrics, while requiring only 0.8–2.7 seconds per image on an A100.

Low-data image restoration.

We understand the request for three recent baselines for each data modality and inverse-problem setting. However, established benchmarks for learning unknown degradations from only 128 paired images remain limited. We therefore utilize ControlNet and LoRA as a representative low-data adaptation baselines and additionally include DEFT using its official implementation on the three supported tasks (Figure 5, Table 11). All methods use the same 128 training pairs. We also explicitly indicate that DEFT is evaluated without restart refinement.

For ImageNet-512, we follow the existing benchmark on low-data 8×8\times super-resolution. This benchmark currently covers only one restoration task in the low-data setting, starting from 25 training pairs. Here, we compare against all three available CSE-based configurations: CSE, DPS+CSE, and DMAP+CSE, wherever results are provided (Figure 3, Table 13).

Speech restoration.

A similar difficulty arises for speech tasks, where published methods often use different datasets, degradation settings, and evaluation protocols. This is particularly challenging for the newer EARS-v2 benchmark. Nevertheless, we extended the full-data comparisons with an additional available baseline (Table 1). Importantly, our primary goal in the full-data regime is to verify that LSA remains competitive beyond limited supervision. We therefore include direct posterior training as the closest controlled comparison, using the same training data and backbone.

Restart refinement.

We agree that restart refinement is orthogonal to the proposed method. To isolate its contribution, we added LSA results without restarts throughout the main comparisons (Tables 1–2, Figure 3). Furthermore, Figure 6 presents a dedicated ablation across the seven ImageNet-256 tasks, examining the impact of removing restarts, both with and without matching the total NFE, as well as the contribution of guidance scaling.

The results show that removing restarts generally degrades LPIPS and FID, with similar trends under matched-NFE comparisons, showing that the improvements are not solely due to additional network evaluations.

Moreover, we justify the use of restarts by the fact that our model is very lightweight during inference; additional restart-based refinement does not add a significant amount of absolute inference time, whereas training-free methods take significantly more time per image on an NVIDIA A100 GPU.

W3. Hyperparameter selection and its sensitivity

We clarified the hyperparameter search protocol and report the selected configurations (Appendix C, Tables 6–7).

To investigate hyperparameter sensitivity, we first conducted an ablation of inference refinements across all seven image tasks (Figure 6). We then selected two representative tasks with different sensitivity profiles, 4×4\times super-resolution and motion deblurring, for a more detailed analysis (Appendix C.1, Figures. 7–8). Keeping the trained LSA networks fixed, we varied total NFE, restart time τ\tau, and the guidance weight ζ\zeta around the configurations selected in validation. These experiments quantify how restoration quality changes when inference hyperparameters deviate from their optimized values.

Importantly, all baselines that we train ourselves, including ControlNet, LoRA, and direct posterior models, undergo the same validation-based hyperparameter search. For external pretrained baselines, we use the authors' original inference configurations whenever available. These protocols are detailed in Appendix B.4.

W4. Related work and citations

We thank the reviewer for pointing out the missing references. We corrected a minor naming typo for Stream.FM and updated the bibliographic references for DiffPhase, Stream.FM, and AnyEnhance.

We have also added several relevant citations of works on inverse problems.

W5. Task-specific adaptation

We agree that training-free methods such as DPS have the important advantage of requiring no task-specific model adaptation. LSA offers a different trade-off: it trades initial training and validation costs, for reduced computation during inference. Moreover, our method can be applied to a wide range of real-world problems where the degradation operator is unavailable.

We therefore view LSA as complementary to training-free posterior samplers and not as a universal replacement. We explicitly acknowledge task-specific training as a limitation in Section 6.

I thank the authors for their clarifications and additional experiments, which have strengthened the paper and addressed my concerns. In light of the rebuttal, I am increasing my score by one point.

AuthorsOpenReview ↗

Big thanks!

We thank the reviewer for their positive feedback and careful reconsideration of our work. We greatly appreciate the constructive comments, which helped us strengthen the experimental evaluation and improve the manuscript!

Reviewer_whkVOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Likelihood Score Approximation (LSA), which adapts a frozen generative prior to inverse problems by learning an additive observation-conditioned correction from paired data. The framework supports score and velocity parameterizations, deterministic and stochastic sampling, and empirical prior swapping. Experiments cover speech and image restoration under limited paired supervision and training with self-generated pairs. Results show strong low-data performance and substantially fewer sampling steps than several known-operator posterior-sampling baselines.

Critical strengths
  1. LSA provides a modular framework that separates the frozen prior from observation-dependent adaptation, supports both score and velocity formulations, and does not require operator access during inference. Its applicability to both unknown degradations and self-generated training pairs is practically useful.
  2. The evaluation spans multiple speech and image tasks, with strong results under limited supervision. Comparisons using identical fixed training pairs, together with parameterization and architecture ablations, provide meaningful evidence for the method’s versatility.
Critical weaknesses
  1. The methodological advance over DEFT needs sharper empirical isolation. Both learn an additive posterior–prior correction; LSA primarily changes the conditioning interface and extends the parameterization. Existing comparisons do not isolate these changes from differences in backbone, capacity, and sampling configuration, making the source of improvement difficult to assess.
  2. The strength of the efficiency and quality comparisons is limited by differing pretrained priors and evaluation protocols. NFE reductions are substantial, but do not directly establish end-to-end speedups after accounting for auxiliary-network evaluations and task-specific training. ImageNet-256 results also rely on only 100 test images without reported uncertainty estimates, making FID-based rankings less conclusive.
Questions for authors
  1. Can you provide a controlled comparison with DEFT using the same prior, comparable correction-network capacity, and matched sampler tuning to isolate the benefit of LSA’s conditioning design?
  2. What are the measured inference latency and total adaptation costs, including synthetic-data generation? Can the principal image results be supported by repeated runs or a larger evaluation set?
  3. Equation (10) shows that LSA absorbs the original prior’s approximation error. Beyond the reported swapping experiments, what diagnostics indicate when replacing the prior will preserve or improve restoration quality?
1 comment

We thank the reviewer for their careful assessment and for recognizing the modularity, versatility, and data efficiency of LSA. We particularly appreciate the constructive suggestions regarding controlled comparisons and computational costs, which motivated several additional experiments and helped us strengthen the revised manuscript.

Q1. Controlled DEFT comparison

To better isolate the methodological differences between LSA and DEFT, we added a dedicated ablation study (Appendix E, Table 12, Figure 11).

We considered ImageNet-256 4×4\times super-resolution with 128 fixed real training pairs. We kept the frozen PixelDiT prior, correction-network architecture, and training configuration identical, varying only the conditioning inputs: the observation yy (the only original condition of LSA), the prior-derived estimate x^0\hat{x}_0, and the log-likelihood gradient g=∇x^0log⁡p(y∣x^0)g=\nabla_{\hat{x}_0}\log p(y\mid\hat{x}_0). We evaluated different combinations of these inputs, including the DEFT-style parameterization with an additional residual guidance term (NN1ψ(xt,x^0,g,t)+NN2ψ(t)g\mathrm{NN}_1^\psi(x_t,{\hat{x}_0},{g},t)+{\mathrm{NN}_2^\psi(t)g}). Inference hyperparameters were independently optimized for each configuration, and all results were evaluated on 1,000 test images.

We find that the conditioning only on the observation yy achieves substantially better perceptual metrics: FID 34.4234.42 versus 44.0944.09 and LPIPS 0.2320.232 versus 0.2480.248 for the DEFT-style configuration. Conversely, DEFT-style conditioning achieves slightly better PSNR and SSIM (25.5425.54 versus 24.9824.98 and 0.7050.705 versus 0.6910.691, respectively). This shows a trade-off between perceptual quality and distortion metrics, also discussed in the original DEFT paper. Figure 11 additionally provides qualitative comparisons across the conditioning variants.

Q2. Inference and adaptation costs

As requested, we extended our main ImageNet-256 evaluation from 100 to 1,000 test images. The DAPS benchmark provides an extended set of 1,000 images, although the original evaluation uses only the first 100 due to the high computational cost of posterior-sampling methods. We updated Table 2, Figure 5, and all main experimental results accordingly.

We also expanded Table 2 with several recent baselines and measured end-to-end inference latency instead of NFEs for all methods on a single NVIDIA A100 GPU.

We also added training dynamics and hyperparameter-search costs results (Appendix H, Figures 12–13) for both self-generated and 128-real-pair training regimes. These experiments report wall-clock costs over 150 training epochs and 150 hyperparameter trials. Importantly, synthetic samples are generated incrementally during training (before each epoch), and their generation time is already included in the reported total training costs.

Notably, unlike some posterior-sampling baselines whose runtime increases substantially for nonlinear operators, LSA maintains relatively consistent inference latency across tasks, as the degradation operator is not required during sampling.

Q3. Prior-swapping diagnostics

We thank the reviewer for this important question. As equation 10 shows, LSA learns a correction relative to the original frozen prior including its approximation error. Replacing the prior therefore changes the combined posterior field and it is not clear whether the trained LSA will remain compatible with a new prior.

To address this, we investigated if the original LSA training objective, presented in equation 9, could serve as a diagnostic tool for prior compatibility.

We evaluated this idea in two prior-swapping scenarios and added the corresponding analyses to Appendix J (Figures 15–16, Table 16). Specifically, we keep the LSA network fixed, replace the prior model and evaluate the resulting objective on a small set of paired examples, using the identical time and noise samples across prior models.

We evaluated this diagnostic for both speech enhancement and image Gaussian deblurring. In speech, larger LSA objective values generally correspond to poorer restoration quality, particularly for the out-of-domain prior. In image restoration, lower objective values are associated with better FID, although PSNR and SSIM show a different trend. Overall, these results show that the LSA objective can provide useful information about prior compatibility.

Reviewer_sFuzOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

"Livin' on a prior" proposes to extend a pretrained unconditional generative model into an observation-conditioned posterior model by keeping the pretrained prior model fixed and focussing on approximating the likelihood score from paired data. The authors motivate their approach by stating that different inverse problems may share the same prior over clean data (which can be represented by an unconditioned generative model) but might have differing likelihoods induced by their respective observation processes. They thus study if it is possible to keep the clean-data prior fixed (,i.e., by freezing the weights of the generative model) and only learn the observation-dependent component. They introduce two variants of the method, depending on the availability of the observation model A. In cases where A is unknown, real paired data samples are required to train the proposed likelihood score approximater. For applications where A is known, no task specific training data is needed as training pairs are generated on the fly by making use of A. The authors additionally provide both a deterministic and stochastic posterior sampling method based on the learned LSA together with the frozen prior.

Critical strengths
  • The method is well motivated and includes a deterministic as well as stochastic posterior sampling approach. It covers two different setups, i.e., the setting where the observation model A is known and the setting where it is unknown and real paired training samples are required.
  • The experimental results are extensive, covering several speech as well as imaging inverse problems. The authors also test exchanging the frozen prior model while keeping the same LSA - highlighting the applicability of the approach even for changing priors.
Critical weaknesses

While given a lot of emphasis in Section 2, the stochastic posterior sampling approach is given little attention in the experimental evaluation and the deterministic setup is chosen as the default.

Questions for authors
  • The authors test the proposed method for known degradation operators by generating training pairs on the fly. For the experiments where in addition to the known degradation operator there exist real training pairs, a comparison between the two settings for an otherwise identical setup would be interesting. How much worse/better does the same approach perform for self-generated data in comparison to using the real pairs?
  • Can you find more applications where stochastic sampling is used rather than the deterministic version of the proposed method. This would make the paper more convincing and balanced.
3 comments

We thank the reviewer for their positive assessment of LSA's unified formulation of known/unknown operators. We appreciate the constructive suggestions, which motivated us for two additional ablation studies in the revised manuscript.

Q1. Real versus synthetic pairs

We added two controlled experiments comparing real and self-generated training pairs (Appendix L, Tables 17,18).

We used fixed pools of either 128 or 1,280 clean samples, keeping training and inference settings identical within each comparison, and varied the source of clean samples: real ImageNet images or samples generated by the pretrained prior (self-generated).

For 4×4\times super-resolution (Table 17), we observe no substantial difference between the two training sources across all metrics and both data budgets.

We additionally investigate box inpainting (Table 18), which has shown a larger gap between self-generated and 128 fixed real pairs in our original evaluation. Since this task involves randomly positioned masks, we compare fixed observations with observations regenerated during training for both: real and generated clean data.

Interestingly, with fixed observations, self-generated data initially outperforms real data (FID 59.8659.86 vs. 63.1263.12, LPIPS 0.1920.192 vs. 0.1980.198). However, allowing observations to vary during training substantially improves both regimes and nearly eliminates this gap (FID 56.0756.07 vs. 56.1656.16, LPIPS 0.1880.188 for both).

While increasing the clean-data pool to 1,280 samples further improves performance, neither source consistently outperforms the other.

Overall, these experiments show that self-generated training data can achieve comparable restoration quality to real data, at least for in-domain training scenarios.

Q2. Extended stochastic evaluation

We extended the stochastic sampling evaluation to all seven image restoration tasks (Appendix K, Figure 18). We investigated stochasticity levels ρ=0.2\rho=0.2 and ρ=0.4\rho=0.4, performed additional hyperparameter searches, and allowed larger NFE budgets to account for the higher computational demands of stochastic sampling.

We found no consistent impromenets from stochastic sampling over deterministic sampling. While it can improve FID or DNN-based MOS on some tasks, these gains are not consistent and typically require substantially more NFEs. Thus, deterministic sampling is our default, and we explicitly acknowledge the limited practical benefit of stochastic sampling in the revised limitations (Section 6).

Thank you very much for providing a detailed rebuttal and very interesting additional results. All my suggestions have been addressed and I will therefore keep "clear acceptance" as suggested score.

AuthorsOpenReview ↗

Big thanks!

We sincerely appreciate the reviewer's positive feedback and the follow-up. We are glad that the additional experiments addressed the reviewer's suggestions and clarified the findings.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.