- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes ANaLOG, a native-latent guidance method for solving inverse problems with latent diffusion models. Building on SILO, which guides sampling with a learned latent forward operator, it interprets that guidance as an isotropic Gaussian likelihood on the encoded measurement and extends it with a learned, input- and time-dependent diagonal covariance (in the spatial or DCT basis), trained by Gaussian negative log-likelihood. Guidance then uses a variance-weighted residual. A theoretical analysis in a linear Gaussian model with deconvolution operators shows that the true latent likelihood is anisotropic and that isotropic approximations incur a strictly positive KL gap. Experiments on FFHQ and ImageNet show improved perceptual quality over SILO on most tasks at essentially the same cost, with substantially faster sampling than decode-and-guide methods.
Critical strengths
- The paper is easy to read and frames the work clearly relative to covariance-aware guidance for pixel-space diffusion and to decode-and-guide and native-latent methods for latent diffusion models. Recasting SILO as an isotropic Gaussian likelihood is a clean way to show the need for a direction-wise covariance.
- Adding the learned covariance improves perceptual quality over SILO on most tasks and on both datasets at essentially the same inference cost, while remaining substantially faster than decode-and-guide methods.
Critical weaknesses
W1. The ablation does not support the claim that anisotropy, input dependence and time dependence are each needed.
- The "time-independent" variant freezes the variance computed at the first sampling step. This seems to remove input dependence as well as time dependence, since the sample is pure noise at that point and the paper notes that early variance maps are relatively uniform. It therefore does not isolate the role of time dependence.
- The intermediate variants give mixed results. For example, the time-independent variant improves FID over SILO but worsens LPIPS, and the theory-predicted covariance worsens FID. Only the full model improves all metrics, and only one task is ablated. A cleaner ablation, such as a variance head trained without the time input or a time-dependent but input-independent covariance, ideally on more than one task, would better support this central claim.
W2. Limited novelty and limited theoretical support.
- Methodologically, the contribution is SILO plus a heteroscedastic variance head trained with a standard Gaussian NLL. The method also inherits the limitations of native-latent guidance, as the authors acknowledge. This includes task-specific training, and measurements that must be image-like.
- The theory covers a restricted case with a linear Gaussian model with a linear autoencoder and deconvolution operators, which the authors present as a motivating example. Within it, the main results are standard. The sufficiency argument does not apply to inpainting or JPEG, and the theory-predicted covariance gives only mixed improvements over SILO in the ablation, so the theory does motivate covariance modeling but offers limited guidance for the method used in practice.
Questions for authors
- How should the basis (spatial vs. DCT) be chosen for a new operator? The better choice appears to vary across tasks; is there some principled criterion, or could an input-adaptive basis be learned?
- Figure 1 (right) is hard to interpret. The ellipse is labeled but is drawn around in latent space, alongside the manifold. Please make this figure more clear.
- Cleaner covariance ablations as described in W1, ideally on more than one task.
- A comparison with ReSample, which is cited but not compared, as a standard latent-diffusion baseline.
3 comments
Rebuttal Response [2 / 2] (Questions 1-4)
“How should basis be chosen?” (Question 1)
We believe that the principled criterion is to match the covariance structure to the structure of the forward operator. Many inverse problems have forward operators that have frequency-domain structure (such as low pass filtering for blurring and downsampling), which is why we believe the DCT-diagonal covariance is generally a good choice for the covariance model. Indeed, in our experiments, across all tasks, ANaLOG-DCT yields competitive performance, often outperforming ANaLOG especially for JPEG decompression. The idea of learning an input-adaptive basis is a great idea, and the natural generalization is to parameterize the covariance as where and are both learned, perhaps with low-rank structure. The main constraint is that this introduces computational cost both at train time and inference time since we must invert this covariance and efficient representations must be developed. Nonetheless, this represents a fascinating direction for future work.
“Figure 1 is hard to interpret” (Question 2)
Thank you for the feedback. We have updated the figure to be more interpretable and we have explained its significance in the caption.
“Comparison to Resample” (Question 4)
We additionally compare against ReSample [2], a widely used decode-and-guide LDM inverse solver, on JPEG decompression. ReSample’s hard data-consistency procedure obtains slightly better FID and PSNR, whereas ANaLOG-DCT obtains substantially better LPIPS. Most importantly, ANaLOG-DCT is approximately 24× faster, demonstrating a favorable quality–efficiency trade-off.
| Method | Time (s) | FID ↓ | LPIPS ↓ | PSNR ↑ |
|---|---|---|---|---|
| ReSample | 576 | 25.7 | 0.421 | 26.96 |
| ANaLOG-DCT | 24.1 | 27.3 | 0.326 | 26.26 |
[2] B. Song et al., “Solving Inverse Problems with Latent Diffusion Models via Hard Data Consistency,” ICLR 2024.
I thank the authors for the response. Most of my concerns have been addressed. I still hold my position on the limited novelty and theoretical contribution, thus, I will maintain my score.
Rebuttal Response [1 / 2] (Weaknesses 1-2)
We thank the reviewer for their insightful feedback. We now respond in detail to each weakness and question.
We include a more thorough ablation study of covariance structure on both Gaussian deblurring and inpainting below. The ablation study in the paper used a small measurement noise, which we believe led to the mixed performance between the covariance variants that the reviewer pointed out. Specifically, with a convolutional operator the measurement noise determines which frequencies remain recoverable, and at the small noise used in the paper's ablation the posterior is tight enough that all covariance variants behave similarly. Below, we use measurement noise σy=0.08 for deblurring.
Following the reviewers' suggestions we add two rows: (a) a variance head trained without the timestep conditioning input, and (b) an isotropic variant that, at each step, replaces ANaLOG's predicted variance by its spatial mean, so that it retains the input and time dependence of ANaLOG but discards the anisotropy. Removing the time input largely degrades performance, demonstrating the importance of time-dependence of the covariance. Replacing ANaLOG with its spatial mean is the strongest alternative, as expected because it keeps the input and time-dependence. ANaLOG still yields the best perceptual quality on both tasks, indicating that input-dependence, time-dependence, and anisotropy are important. We have updated the paper to include this ablation study.
Gaussian deblurring
Center inpainting
Please refer to global comment.
We would like to clarify the novelty of our theoretical work. Notably, there is no existing work that studies native-latent guidance, so our work represents the first formalization. In our model, we study the approximations of native-latent guidance and when they are justified (Proposition 7). Further, we provide an exact characterization of approximation error for SILO and for our method (Theorem 8). Relative to prior latent diffusion posterior-sampling theory [1], our spectral data model further allows us to isolate the role of covariance structure and motivate our method. We believe these results are not standard and shed light on the importance of covariance modeling for native-latent guidance.
[1] L. Rout et al., “Solving Linear Inverse Problems Provably via Posterior Sampling with Latent Diffusion Models,” NeurIPS 2023.