- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
This paper addresses the problem of setting time-dependent guidance weights in training-free posterior sampling (Plug-and-Play) algorithms for linear inverse problems using pretrained diffusion or flow-matching models. Rather than relying on heuristic schedules or expensive iterative optimization, the authors make the observation that the conditional denoising score-matching (diffusion) and conditional flow-matching objectives can be formulated as least-squares regression problems. Consequently, optimizing the balance between the unconditional prior field and the measurement-guidance term reduces at each step to a two-dimensional linear least-squares problem.
To account for exposure bias and the distribution shift between forward diffusion marginals and actual sampling trajectories, the authors propose SIMCA (SIMulation-based CAlibration), a greedy, simulation-based offline calibration scheme that sequentially optimizes guidance weights using a single minibatch of synthetic trajectories. Experiments across diffusion models (FFHQ, ImageNet) and flow-matching models (CelebA, AFHQ-Cat) demonstrate that SIMCA matches or outperforms competitive training-free baselines, collapses performance differences among various guidance formulations (e.g., DPS, GDM, TMPD), and enables up to a reduction in sampling steps (1000 down to 50) without severe perceptual loss.
Critical strengths
- Conceptual Simplicity and Elegance: Formulating the guidance weight tuning problem through the lens of conditional score matching and conditional flow matching targets (Eqs. 4, 10, 12) is mathematically sound, elegant, and provides a principled ground for what was previously mostly treated as ad-hoc empirical tuning.
- Computational Efficiency: Because the objective is quadratic in , each step requires solving an elementary linear system. Calibration takes only seconds to a few minutes offline (Table 5) and avoids backpropagation through ODE/SDE trajectories (unlike diffusion optimal control methods).
- Unified Perspective Across Model Families: The paper provides a unified notation and formulation encompassing both diffusion models and Gaussian flow-matching backbones. The fact that SIMCA operates out-of-the-box on flow velocity fields without score conversion highlights the generality of the principle.
- Acceleration via Discretization-Aware Calibration: Calibrating weights specifically for reduced step budgets () allows DDPM to retain high reconstruction quality and even achieve higher PSNR/SSIM, providing a practical path toward fast posterior sampling.
Critical weaknesses
-
Statistical Reliability of Low-Sample Distribution Metrics (FID on ):
- While I acknowledge that evaluating on 100 images follows the benchmark protocol of DAPS (Zhang et al., 2025) to allow direct comparison with published baselines, computing Fréchet Inception Distance (FID) on only 100 samples is statistically ill-conditioned. Estimating a 2048-dimensional covariance matrix from 100 samples produces a rank-deficient matrix () with extreme sample variance and severe positive bias.
- Small differences in FID (such as those in Tables 1 and 6) under this regime are within the noise margin and cannot reliably support claims of superior distribution coverage.
- The authors should consider reporting sample-efficient distribution metrics designed for smaller sample sizes, such as CMMD (CLIP Maximum Mean Discrepancy, Jayasumana et al., 2024), to confirm that distribution-level conclusions hold.
- Clarification on text: In line 911, the text states "and FID [...] averaged over individual images". Please clarify this wording, as FID is a distribution-level metric and cannot be evaluated or averaged per-instance.
-
Structural Ceiling on Ill-Conditioned Operators:
- The paper acknowledges in lines 374–377 that SIMCA underperforms on deblurring and denoising. Specifically, on ImageNet Gaussian deblurring, SIMCA trails DDNM by almost 2.8 dB (25.28 dB vs. 28.06 dB) and has a substantially worse FID (121.85 vs. 81.43).
- This demonstrates that scalar-tuning of Tweedie guidance cannot overcome the fundamental limitation of isotropic score approximations on ill-conditioned / low-pass operators, where subspace/null-space decomposition (e.g., DDNM) remains strictly superior.
-
Lack of Variance Analysis for the "Single-Image" () Calibration Claim:
- In Section 5.3 and Table 5, the authors assert that a single image () is sufficient for calibration. However, this is shown for only one specific sample.
- If is drawn as a low-texture or low-contrast image (e.g., flat sky or background), the conditioning of the Gram matrix in Algorithm 1 will differ drastically from an image with high-frequency edges. To substantiate the claim that is universally sufficient, the authors should report the mean and standard deviation over 10–20 independent single-image draws.
Questions for authors
-
Behavior Under Varying Noise Levels ():
- In the paper, the measurement noise level is kept fixed at . How do the optimal trajectory weights shift when evaluating across different noise regimes, such as ?
- Does scale inversely with noise variance (e.g., proportional to as classical Bayesian likelihood weighting would suggest), and does stay closer to as noise increases to regularize the reconstruction?
-
Robustness to Operator Misspecification (Semi-Blind Settings):
- In real-world applications, acquisition parameters are rarely known with perfect accuracy. How sensitive is SIMCA if there is a mismatch between the calibration operator and the test operator?
- For example, if guidance weights are calibrated using a Gaussian blur kernel with standard deviation , how significantly does reconstruction quality degrade if the test image was blurred with , or if the true noise level is instead of ?
-
Transferability Across Datasets and Backbones:
- Do the calibrated schedules primarily capture the geometry of the measurement operator , or are they sensitive to the specific dataset prior and network architecture?
- For a fixed operator (e.g., super-resolution), could the authors:
- Compare the schedules: Overlay the calibrated curves obtained across different datasets (e.g., FFHQ vs. ImageNet) and backbones on the same plot (analogous to Figure 1) to visually show whether the trajectories align or diverge.
- Evaluate the numbers: Report quantitative reconstruction metrics (PSNR, SSIM, LPIPS) when transferring weights directly (e.g., applying FFHQ-calibrated weights to ImageNet, or across different checkpoints) without recalibrating.
- This would clarify whether practitioners can reuse pre-computed schedules for a given operator, or if calibration must strictly be repeated for every new dataset and model.
-
Comparison with Baseline Guidance Schedules (in Appendix):
- In Figure 1, the calibrated schedules are shown in isolation. Could the authors overlay on the same axes (e.g., in an appendix figure) the effective schedules used by key baselines? Specifically, plotting the empirical average of the residual-normalized weight of DPS ( with ) and the analytical weights of TMPD, -GDM, or PnP-Flow.
- Visually comparing these curves would help explain where existing heuristics fail (e.g., do they over-guide or under-guide as ?) and provide intuitive support for the claim in Table 4 that baselines mostly suffer from poor scaling.
-
Applicability to Non-Linear Inverse Problems:
- The method is strictly formulated for linear measurement models (). How does the least-squares formulation in Equations 10–12 behave when the forward operator is non-linear (e.g., phase retrieval , non-linear camera tone curves, or JPEG quantization)?
- Does non-linearity in cause instability in the 2D linear regression step, or can the framework be adapted to non-linear inverse problems without modifications?
8 comments
Response to Reviewer TM7j (2/5)
"In the paper, the measurement noise level is kept fixed at . How do the optimal trajectory weights shift when evaluating across different noise regimes, such as ? Does scale inversely with noise variance (e.g., proportional to as classical Bayesian likelihood weighting would suggest), and does stay closer to 1.0 as noise increases to regularize the reconstruction?"
We calibrated the weights on FFHQ (DDPM, steps, calibration images) for noise levels from to , on box inpainting, random inpainting and super-resolution. The likelihood term is taken at the true noise level, , so that classical Bayesian weighting corresponds to . Table B reports at selected steps.
Likelihood weight. In these experiments, the classical scaling is observed only toward the end of the trajectory. Over roughly the first steps, is approximately proportional to (the ratio varies by less than 3% for ), so the effective guidance is nearly independent of the noise level. In the last steps, becomes of order one, closer to the classical weighting, and this happens earlier when the noise is larger.
Prior weight. appears largely insensitive to the noise level: for all tasks and noise levels except in the last three steps, with a deviation that increases slightly with . We therefore do not observe moving closer to as the noise increases.
Table B. Calibrated likelihood weight on FFHQ (DDPM, 50 steps, 10 calibration images). Step is the first, noisiest step.
| Task | ||||||
|---|---|---|---|---|---|---|
| Box inpainting | ||||||
| Random inpainting | ||||||
| SR () | ||||||
These results suggest that the calibrated schedule depends on the noise level in a fairly regular way: the prior weight changes little, and the likelihood weight mainly changes in the last steps. We will add the corresponding figures, with the full schedules for all noise levels, to the revised paper.
Response to Reviewer TM7j (3/5)
"In real-world applications, acquisition parameters are rarely known with perfect accuracy. How sensitive is SIMCA if there is a mismatch between the calibration operator and the test operator? For example, if guidance weights are calibrated using a Gaussian blur kernel with standard deviation , how significantly does reconstruction quality degrade if the test image was blurred with , or if the true noise level is instead of 0.05?"
We ran the two cases suggested by the reviewer on FFHQ (DDPM, steps, test images). In both, the sampler is left untouched: it uses the weights calibrated for the nominal model and evaluates its likelihood term with the nominal operator and noise level. Only the observations change, with the sampling noise shared across settings.
Misspecified noise level. The sampler assumes , while the true noise level ranges from to (Table C). For deviations of ±20%, the metrics change little (at most dB and LPIPS). The sensitivity is asymmetric: a lower true noise level does not degrade the reconstruction, whereas a higher one does. For the reviewer's example ( instead of ), PSNR decreases by dB on box inpainting and dB on random inpainting, and LPIPS increases by and .
Misspecified blur kernel. The sampler assumes a Gaussian kernel of standard deviation , while the observations are blurred with . PSNR decreases from to dB, SSIM from to , and LPIPS increases from to .
Table C. Misspecified noise level on FFHQ. The sampler always assumes .
| Task | True | PSNR | SSIM | LPIPS | CMMD |
|---|---|---|---|---|---|
| Box inpainting | 22.95 | 0.855 | 0.103 | 0.195 | |
| 22.93 | 0.851 | 0.103 | 0.151 | ||
| (matched) | 22.90 | 0.846 | 0.106 | 0.151 | |
| 22.87 | 0.837 | 0.112 | 0.273 | ||
| 22.76 | 0.807 | 0.140 | 1.149 | ||
| 22.57 | 0.746 | 0.179 | 1.537 | ||
| Random inpainting | 33.89 | 0.937 | 0.057 | 0.086 | |
| 33.26 | 0.924 | 0.062 | 0.044 | ||
| (matched) | 32.91 | 0.916 | 0.069 | 0.048 | |
| 32.19 | 0.896 | 0.084 | 0.071 | ||
| 30.90 | 0.854 | 0.120 | 0.247 | ||
| 29.59 | 0.806 | 0.159 | 0.890 |
In these experiments, SIMCA degrades gradually under moderate misspecification and appears more sensitive to an underestimated noise level than to an overestimated one. We will add these results to the revised paper.
Response to Reviewer TM7j (1/5)
We thank the reviewer for the detailed feedback. Please find our answers below.
"The authors should consider reporting sample-efficient distribution metrics designed for smaller sample sizes, such as CMMD (CLIP Maximum Mean Discrepancy, Jayasumana et al., 2024), to confirm that distribution-level conclusions hold."
We computed CMMD for Table 4 of the paper (see below): it preserves the ordering given by FID, with SIMCA improving on the simulation-free variant for all three measurement terms. We will add CMMD to Tables 4 and 6 of the paper in the revised version.
"Clarification on text: In line 911, the text states "and FID [...] averaged over individual images". Please clarify this wording, as FID is a distribution-level metric and cannot be evaluated or averaged per-instance."
The wording at line 911 is incorrect: PSNR, SSIM and LPIPS are averaged over images, while FID is computed on the set of 100 reconstructions. We will correct this sentence.
"To substantiate the claim that N=1 is universally sufficient, the authors should report the mean and standard deviation over 10–20 independent single-image draws."
We ran this experiment on FFHQ (DDPM, 50 steps, σm=0.05) for Gaussian deblurring, box inpainting and random inpainting. We calibrate 20 schedules, each on a different single image, and reconstruct the same 100 test images with each of them. The observations and the sampling noise are shared across runs, so the calibration image is the only source of difference. Table A reports the mean and standard deviation of the test metrics over these runs.
Table A. Sensitivity of the test metrics to the calibration image on FFHQ (DDPM, 50 steps, σm=0.05, 100 test images): mean ± std over 20 runs, each calibrated on a different single image.
Across the single-image calibrations, the standard deviation is below 0.1 dB in PSNR, at most 0.002 in SSIM and at most 0.003 in LPIPS. For Gaussian deblurring and box inpainting, the mean PSNR is also within 0.02 dB of the value obtained with the full calibration set of 100 images.
In these settings, a single calibration image is sufficient, and the choice of that image has little influence on reconstruction quality. We will add this table to the revised paper.