- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper addresses the computational overhead of enforcing measurement consistency in latent diffusion and flow-based inverse solvers, particularly for high-resolution images and high-dimensional videos. Existing methods typically backpropagate pixel-space measurement gradients through the VAE decoder to obtain latent updates, resulting in substantial memory and runtime costs. The authors analyze forward-only alternatives, identifying zeroth-order drift in direct re-encoding and second-order curvature bias in one-sided encoder differences. To address these limitations, they propose Antisymmetric Latent Update (ASLU), which uses a centered difference of two encoder forward passes to approximate the first-order encoder Jacobian response while eliminating zeroth-order drift and even-order curvature terms. Experiments include controlled comparisons with alternative latent-update rules, integration into FlowDPS and STeP, and evaluation against latent inverse solvers on the DM4CT sparse-view CT benchmark. The results demonstrate substantial reductions in GPU memory usage and runtime while maintaining comparable reconstruction quality.
Critical strengths
-
The paper is well-written, clearly structured, and easy to follow. The presentation provides a coherent progression from the motivation and problem formulation to the proposed method and experimental validation.
-
The paper clearly identifies the computational bottleneck of decoder-backward guidance and provides a well-motivated analysis of existing forward-only alternatives. In particular, the analysis of zeroth-order drift and second-order curvature bias offers a clear mathematical justification for the proposed antisymmetric update.
-
The experimental evaluation is comprehensive, covering controlled comparisons of latent-update strategies, high-resolution image reconstruction, high-dimensional video reconstruction, and sparse-view CT. The results demonstrate substantial memory and runtime reductions while maintaining competitive reconstruction quality across different inverse solvers.
Critical weaknesses
The experimental evidence linking ASLU's second-order curvature cancellation to its reconstruction gains could be strengthened. Table 2 shows higher PSNR than one-sided differencing under matched correction rules, with a larger gain in the analytic-backprojection setting. However, the small- and large-correction regimes use different correction procedures, and their actual correction magnitudes across diffusion timesteps are not quantified. Thus, the larger performance gap does not directly isolate the effect of correction magnitude and the associated curvature bias. The supplementary analyses do not fully resolve this: Appendix C.2 reports reconstruction-level hyperparameter sweeps, while Appendix C.4 compares only ASLU against the exact encoder JVP using a random probe direction. A matched-state comparison using actual reconstruction corrections would provide more direct evidence for the proposed explanation.
Questions for authors
Could the authors quantify the correction magnitudes, including the effective encoder probe magnitude , for the two Table 2 regimes across diffusion timesteps? At representative shared intermediate states, could they compare the approximation errors of one-sided differencing and ASLU against the same scaled encoder JVP, , using actual reconstruction corrections? In particular, holding the decoded state, correction direction, and fixed while varying only the correction magnitude would help isolate how the errors scale and assess whether the proposed curvature-based explanation is supported over the correction magnitudes encountered in practice.
3 comments
Rebuttal for Reviewer M21t [2/2]
[Q1-a] "Could the authors quantify the correction magnitudes, including the effective encoder probe magnitude , for the two Table 2 regimes across diffusion timesteps?"
The new App. C.3 (Figure A3(a), Table A9) shows that the two regimes differ by two to three orders of magnitude. The table gives the per-pixel RMS of (median of 10 images, ). Since at our operating point, .
| rule | step 0 | 7 | 14 | 21 | 27 |
|---|---|---|---|---|---|
| Small (FlowDPS rule) | 0.0017 | 0.0015 | 0.0014 | 0.0014 | 0.0013 |
| Large, | 0.60 | 0.11 | 0.050 | 0.033 | 0.019 |
| Large, | 0.62 | 0.13 | 0.063 | 0.044 | 0.027 |
The large corrections start about 400× larger than the small one and remain 15–40× larger at the last step, while the small correction is nearly constant along the trajectory. At , the large corrections decay less (to 0.026 and 0.044 at the last step), so noise keeps them large for longer.
$ $
[Q1-b] "At representative shared intermediate states, could they compare the approximation errors of one-sided differencing and ASLU against the same scaled encoder JVP, , using actual reconstruction corrections? In particular, holding the decoded state, correction direction, and fixed while varying only the correction magnitude would help isolate how the errors scale and assess whether the proposed curvature-based explanation is supported over the correction magnitudes encountered in practice."
The new App. C.3 does exactly this, and both updates follow their predicted orders. We run the FlowDPS sampler (10 images, ). At five steps, we take the decoded state and the actual small, and corrections. Holding the state, the direction and fixed, we vary only and compare each update with the exact from torch.func.jvp in fp32. For the large corrections, the one-sided error falls with slope 1.0 and ASLU's with slope 1.9–2.0, until ASLU reaches the fp32 rounding floor (; Figure A3(b)). On STeP, where the original App. C.4 (now C.6) used a random probe, the same test on an actual reconstruction correction gives slopes 1.00 and 1.98 in fp64 (App. C.3). The effective probe spans 0.6 down to below , covering every magnitude the sampler actually visits. The relative errors at the operating point (one-sided / ASLU, Table A9) are:
| rule | ||
|---|---|---|
| Small | 0.003 / 0.0002 | 0.008 / 0.0002 |
| Large, | 0.14 / 0.04 | 0.42 / 0.19 |
| Large, | 0.24 / 0.09 | 0.70 / 0.34 |
These errors line up with the reconstruction results in Table 3. For the small correction, both errors are below 1%, and the two updates nearly tie (+0.2 to +0.3 dB). For the large corrections, the one-sided error reaches 24–70% and ASLU's gain reaches +1 to +4 dB. Noise roughly triples the one-sided error at , which matches the widening PSNR gap. ASLU's error is lower on 94–100% of (image, step) pairs; the exception, noted in App. C.3, is the earliest probed step, where is largest.
We hope these analyses address your concern and would be glad to provide further breakdowns.
Dear Reviewer M21t,
Thank you again for your thoughtful review and helpful comments on our paper.
We would like to briefly highlight the updates in our response. Following your request for matched corrections, we found and fixed a mismatch in our original Table 2. We then added a matched- experiment in which both updates receive the same correction, and we varied the correction size (Tables 3 and A7). We also measured the correction magnitude on the sampler's own states (Table A9). The small and large regimes differ by two to three orders of magnitude. Holding the state fixed, we measured each update's error (App. C.3, Figure A3). The one-sided error grows as and ASLU's as . These errors line up with the reconstruction results: when the correction is small, both errors are below 1% and the two updates nearly tie. When it is large, the one-sided error reaches 24–70% and ASLU gains clearly.
We sincerely appreciate the time and effort you have already devoted to reviewing our paper. If time permits before the discussion period closes, we would be grateful if you could read through our response. We would welcome any further thoughts and would be happy to clarify any remaining questions.
Best regards,
The Authors
Rebuttal for Reviewer M21t [1/2]
Thank you for a very precise review. Answering your request for matched corrections, we found and fixed a mismatch in our original Table 2. We ran every analysis you asked for (§4.1, Table 3; App. C.2, Table A7; new App. C.3, Figure A3, Table A9). All new and changed text in the revised paper is in blue.
The three analyses tell one consistent story, and together they link ASLU's curvature cancellation to its reconstruction gains. The lift error grows with the correction magnitude, as O(α) for the one-sided update and O(α2) for ASLU, as Eqs. (5) and (7) predict. Where both errors are tiny, the two updates tie; where the one-sided error becomes large, ASLU's reconstruction gain becomes large.
$ $
The new matched-Δx experiment (§4.1, Table 3; App. C.2, Table A7) varies the magnitude within a single procedure, with the correction held identical between the two updates. As the revised §4.1 states, "every one-sided/ASLU pair below receives the same pixel correction Δx at every step, with the same seeds and measurements; only the encoder difference changes." Within the large-correction procedure, we vary the magnitude (K=1, K=3, and K=3 with a doubled step) and the noise level. The gap grows monotonically along both axes (Table A7): at σn=0.2 it is +1.82, +3.99 and +5.03 dB, with ASLU higher on 99–100 of 100 images. For the small FlowDPS correction, the two updates are nearly tied. An excerpt of Table 3 (PSNR↑ / SSIM↑ / LPIPS↓ / FID↓; bold: the better update):
The original Table 2 had paired a K=1 one-sided update with a K=3 ASLU update; matched, the gap at σn=0.03 is +1.05 dB rather than +1.58 dB (App. C.2). The conclusion holds, but the original number was overstated, and we thank you for the question that led us to this. The correction magnitudes and the matched-state comparison you asked for are in our answers to [Q1-a] and [Q1-b] below.