acceptodds
Under review as a conference paper at ICLR 2027

From Decoder Backward to Encoder Forward: Antisymmetric Latent Updates for Diffusion Inverse Solvers

Abstract

High-dimensional and high-resolution inverse problems are difficult to solve with latent diffusion inverse solvers (DIS) because measurement consistency is imposed in pixel space, while the generative prior evolves in latent space. Thus, every sampling step must convert a pixel-space correction into a latent update . Most existing methods perform this conversion by backpropagating through the decoder, which dominates runtime and memory at high resolution; forward-only alternatives such as direct re-encoding avoid this cost but distort the latent prior enough to degrade reconstruction quality. We propose Antisymmetric Latent Update (ASLU), : an antisymmetric difference of two encoder forward passes that vanishes at , preventing accumulated drift, and cancels second-order encoder-curvature bias while preserving the intended first-order Jacobian response. Because ASLU consumes only —never a gradient of the decoder or the measurement operator—the same rule applies whatever DIS mechanism produces the correction. Controlled experiments show that curvature cancellation matters increasingly for larger corrections, while drop-in use in high-resolution image inverse problems and high-dimensional black-hole video reconstruction reduces peak memory by up to and , respectively. On an independent sparse-view CT benchmark against multiple latent DIS methods, ASLU maintains competitive reconstruction quality in a low-memory, short-runtime region not reached by the evaluated baselines.

Primary area:
generative models
Keywords:
Diffusion Inverse Solver, Latent Diffusion Model, Inverse Problem

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_M21tOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the computational overhead of enforcing measurement consistency in latent diffusion and flow-based inverse solvers, particularly for high-resolution images and high-dimensional videos. Existing methods typically backpropagate pixel-space measurement gradients through the VAE decoder to obtain latent updates, resulting in substantial memory and runtime costs. The authors analyze forward-only alternatives, identifying zeroth-order drift in direct re-encoding and second-order curvature bias in one-sided encoder differences. To address these limitations, they propose Antisymmetric Latent Update (ASLU), which uses a centered difference of two encoder forward passes to approximate the first-order encoder Jacobian response while eliminating zeroth-order drift and even-order curvature terms. Experiments include controlled comparisons with alternative latent-update rules, integration into FlowDPS and STeP, and evaluation against latent inverse solvers on the DM4CT sparse-view CT benchmark. The results demonstrate substantial reductions in GPU memory usage and runtime while maintaining comparable reconstruction quality.

Critical strengths
  1. The paper is well-written, clearly structured, and easy to follow. The presentation provides a coherent progression from the motivation and problem formulation to the proposed method and experimental validation.

  2. The paper clearly identifies the computational bottleneck of decoder-backward guidance and provides a well-motivated analysis of existing forward-only alternatives. In particular, the analysis of zeroth-order drift and second-order curvature bias offers a clear mathematical justification for the proposed antisymmetric update.

  3. The experimental evaluation is comprehensive, covering controlled comparisons of latent-update strategies, high-resolution image reconstruction, high-dimensional video reconstruction, and sparse-view CT. The results demonstrate substantial memory and runtime reductions while maintaining competitive reconstruction quality across different inverse solvers.

Critical weaknesses

The experimental evidence linking ASLU's second-order curvature cancellation to its reconstruction gains could be strengthened. Table 2 shows higher PSNR than one-sided differencing under matched correction rules, with a larger gain in the analytic-backprojection setting. However, the small- and large-correction regimes use different correction procedures, and their actual correction magnitudes across diffusion timesteps are not quantified. Thus, the larger performance gap does not directly isolate the effect of correction magnitude and the associated curvature bias. The supplementary analyses do not fully resolve this: Appendix C.2 reports reconstruction-level hyperparameter sweeps, while Appendix C.4 compares only ASLU against the exact encoder JVP using a random probe direction. A matched-state comparison using actual reconstruction corrections would provide more direct evidence for the proposed explanation.

Questions for authors

Could the authors quantify the correction magnitudes, including the effective encoder probe magnitude ∥αΔx∥\|\alpha\Delta x\|, for the two Table 2 regimes across diffusion timesteps? At representative shared intermediate states, could they compare the approximation errors of one-sided differencing and ASLU against the same scaled encoder JVP, ραJE(x)Δx\rho\alpha J_E(x)\Delta x, using actual reconstruction corrections? In particular, holding the decoded state, correction direction, and (α,ρ)(\alpha,\rho) fixed while varying only the correction magnitude would help isolate how the errors scale and assess whether the proposed curvature-based explanation is supported over the correction magnitudes encountered in practice.

3 comments
AuthorsOpenReview ↗

Rebuttal for Reviewer M21t [1/2]

Thank you for a very precise review. Answering your request for matched corrections, we found and fixed a mismatch in our original Table 2. We ran every analysis you asked for (§4.1, Table 3; App. C.2, Table A7; new App. C.3, Figure A3, Table A9). All new and changed text in the revised paper is in blue.

The three analyses tell one consistent story, and together they link ASLU's curvature cancellation to its reconstruction gains. The lift error grows with the correction magnitude, as O(α)O(\alpha) for the one-sided update and O(α2)O(\alpha^2) for ASLU, as Eqs. (5) and (7) predict. Where both errors are tiny, the two updates tie; where the one-sided error becomes large, ASLU's reconstruction gain becomes large.

$ $

[W1] "The experimental evidence linking ASLU's second-order curvature cancellation to its reconstruction gains could be strengthened. Table 2 shows higher PSNR than one-sided differencing under matched correction rules, with a larger gain in the analytic-backprojection setting. However, the small- and large-correction regimes use different correction procedures, and their actual correction magnitudes across diffusion timesteps are not quantified. Thus, the larger performance gap does not directly isolate the effect of correction magnitude and the associated curvature bias. The supplementary analyses do not fully resolve this: Appendix C.2 reports reconstruction-level hyperparameter sweeps, while Appendix C.4 compares only ASLU against the exact encoder JVP using a random probe direction. A matched-state comparison using actual reconstruction corrections would provide more direct evidence for the proposed explanation."

The new matched-Δx\Delta x experiment (§4.1, Table 3; App. C.2, Table A7) varies the magnitude within a single procedure, with the correction held identical between the two updates. As the revised §4.1 states, "every one-sided/ASLU pair below receives the same pixel correction Δx\Delta x at every step, with the same seeds and measurements; only the encoder difference changes." Within the large-correction procedure, we vary the magnitude (K=1K=1, K=3K=3, and K=3K=3 with a doubled step) and the noise level. The gap grows monotonically along both axes (Table A7): at σn=0.2\sigma_n=0.2 it is +1.82, +3.99 and +5.03 dB, with ASLU higher on 99–100 of 100 images. For the small FlowDPS correction, the two updates are nearly tied. An excerpt of Table 3 (PSNR↑ / SSIM↑ / LPIPS↓ / FID↓; bold: the better update):

Δx\Delta x ruleσn\sigma_nOne-sided (Eq. (5))ASLU (Ours, Eq. (6))
Small (FlowDPS rule)0.0327.61 / .751 / .153 / 98.227.83 / .769 / .151 / 104.3
0.227.48 / .734 / .165 / 92.227.69 / .753 / .166 / 96.1
Large, K=3K=3 (main)0.0326.10 / .533 / .130 / 70.027.15 / .634 / .129 / 65.7
0.222.84 / .281 / .158 / 82.926.83 / .613 / .139 / 66.4

The original Table 2 had paired a K=1K=1 one-sided update with a K=3K=3 ASLU update; matched, the gap at σn=0.03\sigma_n=0.03 is +1.05 dB rather than +1.58 dB (App. C.2). The conclusion holds, but the original number was overstated, and we thank you for the question that led us to this. The correction magnitudes and the matched-state comparison you asked for are in our answers to [Q1-a] and [Q1-b] below.

AuthorsOpenReview ↗

Rebuttal for Reviewer M21t [2/2]

[Q1-a] "Could the authors quantify the correction magnitudes, including the effective encoder probe magnitude ∥αΔx∥\|\alpha\Delta x\|, for the two Table 2 regimes across diffusion timesteps?"

The new App. C.3 (Figure A3(a), Table A9) shows that the two regimes differ by two to three orders of magnitude. The table gives the per-pixel RMS of Δx\Delta x (median of 10 images, σn=0.03\sigma_n=0.03). Since α=1\alpha=1 at our operating point, ∥αΔx∥=∥Δx∥\|\alpha\Delta x\|=\|\Delta x\|.

Δx\Delta x rulestep 07142127
Small (FlowDPS rule)0.00170.00150.00140.00140.0013
Large, K=1K=10.600.110.0500.0330.019
Large, K=3K=30.620.130.0630.0440.027

The large corrections start about 400× larger than the small one and remain 15–40× larger at the last step, while the small correction is nearly constant along the trajectory. At σn=0.2\sigma_n=0.2, the large corrections decay less (to 0.026 and 0.044 at the last step), so noise keeps them large for longer.

$ $

[Q1-b] "At representative shared intermediate states, could they compare the approximation errors of one-sided differencing and ASLU against the same scaled encoder JVP, ραJEΔx\rho\alpha J_{\mathcal{E}}\Delta x, using actual reconstruction corrections? In particular, holding the decoded state, correction direction, and ρα\rho\alpha fixed while varying only the correction magnitude would help isolate how the errors scale and assess whether the proposed curvature-based explanation is supported over the correction magnitudes encountered in practice."

The new App. C.3 does exactly this, and both updates follow their predicted orders. We run the FlowDPS sampler (10 images, σn∈{0.03,0.2}\sigma_n\in\{0.03,0.2\}). At five steps, we take the decoded state and the actual small, K=1K=1 and K=3K=3 corrections. Holding the state, the direction and ρα\rho\alpha fixed, we vary only α∈{1,12,…,164}\alpha\in\{1,\tfrac12,\dots,\tfrac1{64}\} and compare each update with the exact JE(x)ΔxJ_{\mathcal{E}}(x)\Delta x from torch.func.jvp in fp32. For the large corrections, the one-sided error falls with slope 1.0 and ASLU's with slope 1.9–2.0, until ASLU reaches the fp32 rounding floor (≈2×10−4\approx2\times10^{-4}; Figure A3(b)). On STeP, where the original App. C.4 (now C.6) used a random probe, the same test on an actual reconstruction correction gives slopes 1.00 and 1.98 in fp64 (App. C.3). The effective probe ∥αΔx∥\|\alpha\Delta x\| spans 0.6 down to below 10−310^{-3}, covering every magnitude the sampler actually visits. The relative errors at the operating point α=1\alpha=1 (one-sided / ASLU, Table A9) are:

Δx\Delta x ruleσn=0.03\sigma_n=0.03σn=0.2\sigma_n=0.2
Small0.003 / 0.00020.008 / 0.0002
Large, K=1K=10.14 / 0.040.42 / 0.19
Large, K=3K=30.24 / 0.090.70 / 0.34

These errors line up with the reconstruction results in Table 3. For the small correction, both errors are below 1%, and the two updates nearly tie (+0.2 to +0.3 dB). For the large corrections, the one-sided error reaches 24–70% and ASLU's gain reaches +1 to +4 dB. Noise roughly triples the one-sided error at K=3K=3, which matches the widening PSNR gap. ASLU's error is lower on 94–100% of (image, step) pairs; the exception, noted in App. C.3, is the earliest probed step, where Δx\Delta x is largest.

We hope these analyses address your concern and would be glad to provide further breakdowns.

Dear Reviewer M21t,

Thank you again for your thoughtful review and helpful comments on our paper.

We would like to briefly highlight the updates in our response. Following your request for matched corrections, we found and fixed a mismatch in our original Table 2. We then added a matched-Δx\Delta x experiment in which both updates receive the same correction, and we varied the correction size (Tables 3 and A7). We also measured the correction magnitude on the sampler's own states (Table A9). The small and large regimes differ by two to three orders of magnitude. Holding the state fixed, we measured each update's error (App. C.3, Figure A3). The one-sided error grows as O(α)O(\alpha) and ASLU's as O(α2)O(\alpha^2). These errors line up with the reconstruction results: when the correction is small, both errors are below 1% and the two updates nearly tie. When it is large, the one-sided error reaches 24–70% and ASLU gains clearly.

We sincerely appreciate the time and effort you have already devoted to reviewing our paper. If time permits before the discussion period closes, we would be grateful if you could read through our response. We would welcome any further thoughts and would be happy to clarify any remaining questions.

Best regards,

The Authors

Reviewer_B9AVOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes ASLU, a method for lifting pixel-space measurement corrections to the latent space of diffusion/flow models by only using forward encoder calls. It replaces expensive decoder backpropagation with a centered difference of two encoder evaluations. In this way, a zero pixel-space correction yields a zero latent-space counterpart, and the quadratic encoder curvature term of a one-sided difference gets canceled. Experiments cover various methods and configurations, integrating ASLU into FlowDPS, STeP, and PSLD on image restoration, black-hole video reconstruction, and sparse-view CT. The reported results show large memory savings and a significant runtime improvement, while the reconstruction quality is comparable to the corresponding decoder-gradient baselines.

Critical strengths
  1. The method targets in a simple way a real computational bottleneck that all the Latent DMs inverse solvers have to deal with. Considering how popular these methods have become and how powerful they are, improvements in memory footprint and runtime across all of them are remarkable. Table 3 reports approximately 64% memory savings at 2K resolution and 76% for black-hole videos; Table A1 reports 85% relative to PSLD on CT. This gains even more importance in tasks where memory determines whether a reconstruction can run at all.
  2. The experimental section tests different priors and guidance mechanisms, and includes the comparison with the one-sided encoder baseline. Gains and quality gaps seem consistent across all these tasks.
Critical weaknesses
  1. Experiments with PSLD need more clarification. Indeed, in Appendix A.3, it is mentioned that only the measurement term is changed, while the gluing one stays untouched, yielding a gradient-free update. The original PSLD method differentiates both terms; thus, more clarification is needed about this difference in the implementation.
  2. The experiments do not isolate the proposed method's benefit convincingly enough. Appendix C.2 uses only one image per task and omits extreme grid values, so it cannot establish robust superiority over a separately tuned one-sided baseline. Moreover, reducing the probe radius while increasing ρ\rho to keep ρα\rho \alpha fixed can reduce the one-sided curvature error; the statement that it cannot be tuned away is too strong. These controls matter because removing cycle drift is already achieved by the one-sided rule, while ASLU's additional contribution is curvature cancellation. This point is, in my opinion, the main reason for which right now the work is not yet in an "accept" status.
Questions for authors
  1. Please provide a complete PSLD+ASLU pseudocode, including the gluing update.
  2. Could you compare the one-sided encoder baseline, ASLU, and exact JVP across a dataset, using the same correction direction at different magnitudes? Keep the first-order step size fixed ρα\rho\alpha for finite differences, while tuning each method separately under this constraint. Please report the complete hyperparameter search results, including excluded settings, the parameters used in Table 2 and variability across images and random seeds.
  3. Could the authors discuss and provide results (even just on a single dataset and task) for methods where the update Δx\Delta x comes instead from a proximal update, as in TReg [1] or LATINO [2]? Since these methods use direct encoding, it would be interesting to check whether the proposed technique can improve their results.

[1] Kim, Jeongsol, et al. "Regularization by texts for latent diffusion inverse solvers." International Conference on Learning Representations. Vol. 2025. 2025.

[2] Spagnoletti, Alessio, et al. "Latino-pro: Latent consistency inverse solver with prompt optimization." 2025 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2025.

5 comments
AuthorsOpenReview ↗

Rebuttal for Reviewer B9AV [1/3]

Thank you for the careful reading, especially of the appendix. You found a real inaccuracy in App. A.3, which we have corrected, and your request to isolate the curvature cancellation led to most of our new experiments. All new and changed text in the revised paper is in blue.

$ $

[W1] "Experiments with PSLD need more clarification. Indeed, in Appendix A.3, it is mentioned that only the measurement term is changed, while the gluing one stays untouched, yielding a gradient-free update. The original PSLD method differentiates both terms; thus, more clarification is needed about this difference in the implementation."

[Q1] "Please provide a complete PSLD+ASLU pseudocode, including the gluing update."

The revised App. A.3 corrects our description and adds the full sampler (Algorithm 1) and a gluing-weight ablation (Table A5). PSLD's gluing term is ∥E(Π(D(z^0),AD(z^0)))−z^0∥\|\mathcal{E}(\Pi(\mathcal{D}(\hat z_0), A\mathcal{D}(\hat z_0)))-\hat z_0\|. With DM4CT's SIRT projection, Π(x,Ax)=x\Pi(x,Ax)=x, so the term reduces to the cycle residual E(x0)−z^0\mathcal{E}(x_0)-\hat z_0. We apply it as the forward-only update γ(E(x0)−z^0)\gamma(\mathcal{E}(x_0)-\hat z_0), with γ=0\gamma=0 in all reported results. The per-step update of Algorithm 1 is

zt−1=zt−1′+ρ⋅12[E(x0+αΔx)−E(x0−αΔx)]+γ(E(x0)−z^0),Δx=Π(x0,y)−x0,z_{t-1} = z'_{t-1} + \rho\cdot\tfrac12\bigl[\mathcal{E}(x_0+\alpha\Delta x)-\mathcal{E}(x_0-\alpha\Delta x)\bigr] + \gamma\bigl(\mathcal{E}(x_0)-\hat z_0\bigr),\qquad \Delta x=\Pi(x_0,y)-x_0,

where zt−1′z'_{t-1} is the DDPM step and x0=D(z^0)x_0=\mathcal{D}(\hat z_0), with no backward pass anywhere. Table A5 shows that dropping the gluing term explains part of our gain over the published PSLD. With γ=0\gamma=0 for both methods, ASLU is on par (25.75 vs 25.57 dB on configuration ii) while using 77% less peak memory than a decoder-only PSLD with the same γ=0\gamma=0 (7.07 → 1.65 GiB; Table A3).

AuthorsOpenReview ↗

Rebuttal for Reviewer B9AV [2/3]

[W2] "The experiments do not isolate the proposed method's benefit convincingly enough. Appendix C.2 uses only one image per task and omits extreme grid values, so it cannot establish robust superiority over a separately tuned one-sided baseline. Moreover, reducing the probe radius while increasing ρ\rho to keep ρα\rho\alpha fixed can reduce the one-sided curvature error; the statement that it cannot be tuned away is too strong. These controls matter because removing cycle drift is already achieved by the one-sided rule, while ASLU's additional contribution is curvature cancellation. This point is, in my opinion, the main reason for which right now the work is not yet in an "accept" status."

The new matched-Δx\Delta x experiment (§4.1, Table 3; 100 images, three seeds) replaces the single-image study as the main evidence, and it shows that the curvature cancellation matters when the correction is large or noisy. Both updates receive exactly the same Δx\Delta x, seeds and measurements, so only the encoder difference changes. An excerpt (PSNR↑ / SSIM↑ / LPIPS↓ / FID↓; bold: the better update):

Δx\Delta x ruleσn\sigma_nOne-sided (Eq. (5))ASLU (Ours, Eq. (6))
Small (FlowDPS rule)0.0327.61 / .751 / .153 / 98.227.83 / .769 / .151 / 104.3
0.227.48 / .734 / .165 / 92.227.69 / .753 / .166 / 96.1
Large, K=3K=3 (main)0.0326.10 / .533 / .130 / 70.027.15 / .634 / .129 / 65.7
0.222.84 / .281 / .158 / 82.926.83 / .613 / .139 / 66.4

For the large correction used by our main FlowDPS results, ASLU is better on every metric, and the gap widens with noise because the one-sided update degrades while ASLU barely changes. For small corrections, the two are practically tied, here and in the new one-sided ablations in STeP and DM4CT (Tables A2, A6). This is what Eq. (5) predicts: the curvature bias "scales as α2∥Δx∥2\alpha^2\|\Delta x\|^2".

On tuning the curvature error away, we agree that the original statement was too strong. The revised §4.1 now reads: "Shrinking α\alpha at fixed ρα\rho\alpha does reduce that bias in exact arithmetic, but α\alpha is fixed per pipeline while ∥Δx∥\|\Delta x\| varies along the trajectory, and finite precision limits how far α\alpha can shrink." The new App. C.3 quantifies both limits on real FlowDPS states. Even at its best α\alpha, the one-sided error remains about 10× above ASLU's. In addition, ∥Δx∥\|\Delta x\| changes by more than 10× along the trajectory, so no single α\alpha fits all steps.

Regarding extreme values, the revised App. C.4 (originally C.2) now shows the full 6×56\times5 grids, including α=0.25\alpha=0.25 and ρ=800\rho=800 (Figure A4). Our original claim that "both methods collapse together" there was wrong: ASLU degrades less (Gaussian deblurring at α=0.25,ρ=800\alpha=0.25,\rho=800: LPIPS 0.206 vs 0.399) and has the lower LPIPS in most cells. The best cells are tied, so on these small-Δx\Delta x tasks ASLU's benefit is lower sensitivity, not a better optimum. Conversely, in DM4CT at α=2.4\alpha=2.4 the probe x−αΔxx-\alpha\Delta x leaves the data range and ASLU drops to 11.9 dB (one-sided: 25.7 dB), a limitation stated in App. A.3 and §5.

Finally, we agree that removing cycle drift is already achieved by the one-sided update, which was suggested in the appendix of Graikos et al. (NeurIPS 2025); the revised §3 now credits it. ASLU's additional contribution is the curvature cancellation, and the experiments above show where it matters.

AuthorsOpenReview ↗

Rebuttal for Reviewer B9AV [3/3]

[Q2] "Could you compare the one-sided encoder baseline, ASLU, and exact JVP across a dataset, using the same correction direction at different magnitudes? Keep the first-order step size fixed ρα\rho\alpha for finite differences, while tuning each method separately under this constraint. Please report the complete hyperparameter search results, including excluded settings, the parameters used in Table 2 and variability across images and random seeds."

The new matched-state analysis (App. C.3, Figure A3, Table A9) does exactly this, and the errors scale as Eqs. (5) and (7) predict. On the FlowDPS sampler's own states (10 images, σn∈{0.03,0.2}\sigma_n\in\{0.03,0.2\}, 5 steps each), we hold the state, the direction and ρα\rho\alpha fixed and vary only α∈{1,…,164}\alpha\in\{1,\dots,\tfrac1{64}\}. Each update is compared with the exact JVP JE(x)ΔxJ_{\mathcal{E}}(x)\Delta x. For large corrections, the one-sided error falls with slope 1.0 and ASLU's with slope 1.9–2.0. At the operating point, ASLU's error is 2–4× lower, on 94–100% of (image, step) pairs.

The complete search results, including the previously excluded settings, are the full grids of App. C.4 (Figure A4), where each update is tuned separately. The parameters used in the matched comparison (Table 3, replacing the original Table 2's one-sided/ASLU rows) are α=ρ=1\alpha=\rho=1 for the large correction and a latent step size of 300 for the small one, "the best of {15,100,300,1000}\{15,100,300,1000\} for both lifts on two held-out images" (App. C.2). In the original Table 2, the large-correction one-sided row used a K=1K=1 correction while the ASLU row used K=3K=3; with matched corrections, the gap at the submission's noise level is +1.05 dB rather than +1.58 dB (App. C.2).

Variability across images is given by the per-image win counts in Table 3 and Table A7. For seeds, the new Table A8 repeats every Table 3 cell with three seeds: ASLU leads in every cell for every seed, and for large corrections the gap varies by at most 0.08 dB across seeds.

$ $

[Q3] "Could the authors discuss and provide results (even just on a single dataset and task) for methods where the update Δx\Delta x comes instead from a proximal update, as in TReg [1] or LATINO [2]? Since these methods use direct encoding, it would be interesting to check whether the proposed technique can improve their results."

We ran this for LATINO (new App. C.7, Table A12). There, re-encoding works better than either update, and the revised Limitations (§5) now state this scope. With the official code and settings (FFHQ, 100 images, three tasks), we replaced z^0←E(xprox)\hat z_0\leftarrow\mathcal{E}(x_{\text{prox}}) by the one-sided update and ASLU with Δx=xprox−x\Delta x=x_{\text{prox}}-x; both are worse (Gaussian deblurring: 28.41 dB vs 26.14 and 24.12). The reason is the sampler: LATINO re-noises z^0\hat z_0 with fresh noise at each of its eight steps and passes nothing else on, so there is no latent trajectory for a drift-free update to preserve. ASLU targets samplers that carry the latent across steps, as FlowDPS, STeP and PSLD do. We did not run TReg.

We hope these additions address your concerns, and we would be glad to provide further results.

Reviewer_Juc6OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The paper addresses a recurring step in latent diffusion inverse solvers: turning a pixel-space correction Δx\Delta x into a latent update Δz\Delta z. Existing methods conventionally backpropagate through the VAE decoder, which is costly at high resolution. Direct re-encoding is cheaper but introduces drift, since E(D(z))≠zE(D(z)) \neq z. The authors propose ASLU, a central difference of two encoder forward passes, ρ⋅12[E(x+αΔx)−E(x−αΔx)]\rho \cdot \frac{1}{2}[E(x+\alpha \Delta x)- E(x-\alpha \Delta x)]. The update vanishes at Δx=0\Delta x=0 and cancels the second-order term that a one-sided difference retains. ASLU is plugged into FlowDPS for 1K-2K images, STeP for black-hole video, and PSLD on the DM4CT benchmark. Peak memory drops by 51-85% across these settings, with reconstruction quality broadly comparable to the original solvers.

Critical strengths
  • The paper is well-written and easy to follow.
  • ASLU is simple but effective. The proposed algorithm needs two encoder forward passes and stores no activation graph. It replaces decoder backpropagation without altering the rest of the sampler. Peak memory falls by 64%~85% for three tasks. Note that reconstruction quality is comparable to the original pipelines in each case.
  • The paper classifies lifts by two properties: whether they require derivate propagation, and whether they vanish at Δx=0\Delta x =0. This view explains why direct re-encoding accumulates drift over sampling steps. It also motivates differencing against a reference encoding instead of replacing the latent. The central difference is demonstrated to converge to the exact encoder JVP, so the update estimates the intended first-order quantity. The resulting rule is easy to adopt in other latent solvers.
Critical weaknesses
  • The paper centers on anti-symmetry and the cancellation of second-order curvature bias. Although the efficiency gains come from being forward-only, a property the one-sided difference shares at identical cost. None of FlowDPS, STeP or DM4CT integrations tests the one-sided variant. Evidence for curvature cancellation is confined to one controlled setting, with a large backpropagation step and untuned (α,ρ)(\alpha, \rho). Under the actual FlowDPS update, the PSNR gain is marginal and FID worsens. Thus, the core claim is not supported where it matters most.
  • The method lacks a principled justification beyond finite differencing. The open question is why the lifted quantity is a good latent update at all. ASLU approximates the encoder JVP, whereas decoder-backward guidance applies the decoder VJP. The paper acknowledges that these operators differ but offers no analysis of when one can replace the other.
  • Since the contribution of the proposed algorithm is mainly practical, the memory baselines are critical. The paper compares only against naive decoder backpropagation. Gradient checkpointing, tiled VAE decoding and reduced precision are standard ways to reduce this cost, and none is evaluated. It is unclear how much of the reported saving would remain in a well-engineered pipeline.
Questions for authors
  • What happens when the one-sided difference replaces ASLU in the FlowDPS, STeP, and DM4CT pipelines, under the same tuning budget? A clear advantage of ASLU there would address my main concern.
  • How does ASLU compare with decoder backpropagation under gradient checkpointing or tiled decoding, in both memory and runtime?
5 comments
AuthorsOpenReview ↗

Rebuttal for Reviewer Juc6 [1/2]

Thank you for the detailed review and for the positive assessment of the method's simplicity. In the revision, we tested the one-sided update inside all three solvers with every hyperparameter shared, compared against memory-engineering baselines, and added a Gauss–Newton view of the encoder lift. All new and changed text in the revised paper is in blue.

$ $

[W1] "The paper centers on anti-symmetry and the cancellation of second-order curvature bias. Although the efficiency gains come from being forward-only, a property the one-sided difference shares at identical cost. None of FlowDPS, STeP or DM4CT integrations tests the one-sided variant. Evidence for curvature cancellation is confined to one controlled setting, with a large backpropagation step and untuned (α,ρ)(\alpha,\rho). Under the actual FlowDPS update, the PSNR gain is marginal and FID worsens. Thus, the core claim is not supported where it matters most."

In the new matched-Δx\Delta x experiment (§4.1, Table 3) and the new one-sided ablations in STeP (Table A2) and DM4CT (Table A6), ASLU is clearly better than the one-sided update for large or noisy corrections and on par for small ones. As derived in §3 (Eq. (5)), the one-sided update retains "an unwanted curvature bias" 12α2HE(x)[Δx,Δx]\tfrac12\alpha^2 H_{\mathcal{E}}(x)[\Delta x,\Delta x], which ASLU cancels (Eq. (7)). In the new Table 3, both updates receive exactly the same Δx\Delta x, seeds and measurements, so only the encoder difference changes. An excerpt (PSNR↑ / SSIM↑ / LPIPS↓ / FID↓, 100 images; bold: the better update):

Δx\Delta x ruleσn\sigma_nOne-sided (Eq. (5))ASLU (Ours, Eq. (6))
Small (FlowDPS rule)0.0327.61 / .751 / .153 / 98.227.83 / .769 / .151 / 104.3
0.227.48 / .734 / .165 / 92.227.69 / .753 / .166 / 96.1
Large, K=3K=3 (main)0.0326.10 / .533 / .130 / 70.027.15 / .634 / .129 / 65.7
0.222.84 / .281 / .158 / 82.926.83 / .613 / .139 / 66.4

You are right about the small FlowDPS correction: the PSNR gain is small and FID is worse with ASLU. The revised App. C.2 now states this as "a limitation of the evidence in the small-correction regime". For the large K=3K=3 correction used by our main FlowDPS results (§4.2.1), ASLU is better on every metric and on nearly every image, by +1 to +4 dB depending on noise. The gap widens with noise because the one-sided update degrades while ASLU barely changes, and it holds across three seeds (Table A8). In DM4CT and STeP, whose corrections are small, the two are practically tied; mean χˉ2\bar\chi^2 on STeP slightly favors the one-sided update because of two videos, and we report this as is (App. A.2). This split is what Eq. (5) predicts, since the curvature term matters only when α∥Δx∥\alpha\|\Delta x\| is large.

We also correct one number. In the original Table 2, the large-correction one-sided row used a K=1K=1 correction while the ASLU row used K=3K=3. With matched corrections, the gap at the submission's noise level is +1.05 dB rather than +1.58 dB (App. C.2). The conclusion holds, but the original number was overstated.

On the shared efficiency, we agree that the forward-only cost is shared with the one-sided update, and we see this as part of our contribution rather than a gap in it. The one-sided encoder difference was suggested in the appendix of Graikos et al. (NeurIPS 2025) as an alternative to decoder backpropagation, but their quantitative experiments used backpropagation; a related form appears in generative anamorphosis. The revised §3 now credits both. Our paper characterizes this update's error (§3, Eq. (5)) and evaluates it inside FlowDPS, STeP and DM4CT. For samplers that carry the latent across steps, as in all three pipelines, ASLU is therefore the right default: it has the same cost and memory savings, it is clearly better when the correction is large or noisy, and it is on par otherwise.

$ $

[Q1] "What happens when the one-sided difference replaces ASLU in the FlowDPS, STeP, and DM4CT pipelines, under the same tuning budget? A clear advantage of ASLU there would address my main concern."

The results are in our answer to [W1] above. In all new comparisons, the two updates share every hyperparameter (§4.1; App. A.2, A.3, C.2). They use the same Δx\Delta x, α\alpha, ρ\rho and seeds. For the small FlowDPS rule, the latent step size was "the best of {15,100,300,1000}\{15,100,300,1000\} for both lifts on two held-out images" (App. C.2), namely 300. ASLU has a clear advantage wherever the correction is large or noisy, which in our pipelines is FlowDPS with its main K=3K=3 correction; in STeP and DM4CT, the corrections are small and the two tie.

AuthorsOpenReview ↗

Rebuttal for Reviewer Juc6 [2/2]

[W2] "The method lacks a principled justification beyond finite differencing. The open question is why the lifted quantity is a good latent update at all. ASLU approximates the encoder JVP, whereas decoder-backward guidance applies the decoder VJP. The paper acknowledges that these operators differ but offers no analysis of when one can replace the other."

We added a Gauss–Newton view to §5 (new paragraph "Why the encoder tangent is a sensible latent update"). It builds on what §5 already stated: ASLU's leading term is JE(x0)ΔxJ_{\mathcal{E}}(x_0)\Delta x whereas decoder-backward guidance uses JD(z0)⊤ΔxJ_{\mathcal{D}}(z_0)^\top\Delta x, so "ASLU should therefore be understood not as a cheap replica of decoder-backward guidance, but as an encoder-native way to lift the same pixel-space correction." The new paragraph, which follows the inexact-Newton view of Graikos et al. (NeurIPS 2025), explains why this encoder-native lift is a sensible update. Because an autoencoder satisfies E(D(z))≈z\mathcal{E}(\mathcal{D}(z))\approx z, differentiating gives

JE(D(z)) JD(z)≈I⇒JEΔx≈JD+Δx.J_{\mathcal{E}}(\mathcal{D}(z))\,J_{\mathcal{D}}(z)\approx I \quad\Rightarrow\quad J_{\mathcal{E}}\Delta x\approx J_{\mathcal{D}}^{+}\Delta x.

ASLU's leading term is therefore approximately the Gauss–Newton step for min⁡Δz∥D(z+Δz)−(x+Δx)∥2\min_{\Delta z}\|\mathcal{D}(z+\Delta z)-(x+\Delta x)\|^2, while JD⊤ΔxJ_{\mathcal{D}}^\top\Delta x is the corresponding gradient step. The two agree only when JDJ_{\mathcal{D}} is well conditioned, which is consistent with the partial alignment (cosine 0.62) and the 81% smaller norm already reported in §5, Table 5. The argument requires E∘D≈Id\mathcal{E}\circ\mathcal{D}\approx\mathrm{Id} at the evaluated points, and we state where it weakens in the same paragraph.

$ $

[W3] "Since the contribution of the proposed algorithm is mainly practical, the memory baselines are critical. The paper compares only against naive decoder backpropagation. Gradient checkpointing, tiled VAE decoding and reduced precision are standard ways to reduce this cost, and none is evaluated. It is unclear how much of the reported saving would remain in a well-engineered pipeline."

[Q2] "How does ASLU compare with decoder backpropagation under gradient checkpointing or tiled decoding, in both memory and runtime?"

In the new efficiency comparison (App. A.1, Table A1; summarized in §4.2.1), these techniques narrow the gap but do not close it. Decoder-backward FlowDPS reaches ASLU's memory only with checkpointing and tiling combined, at about 2.5× its own runtime, and tiling alone runs out of memory at 2K, because the backward pass keeps every tile's activations. Reduced precision is already part of every row, since the FlowDPS VAE runs in fp16 (App. A.1). All rows were measured in one environment, so they should be compared within Table A1 only.

102421024^2: time / mem2K: time / mem
FlowDPS (decoder backward)54 s / 28.6 GB180 s / 48.5 GB
+ checkpointing + tiling137 s / 19.3 GB424 s / 24.0 GB
ASLU20 s / 18.6 GB61 s / 22.7 GB
ASLU + tiling30 s / 16.7 GB86 s / 16.8 GB

We also corrected the DM4CT memory claim (new Table A3; Figure 2 caption). PSLD's implementation also backpropagates through the UNet, so part of the original 85% saving came from that pass. The UNet backward accounts for only about 0.5 GiB. Against a decoder-only PSLD with the same γ=0\gamma=0, ASLU still reduces peak memory by 77% (7.07 → 1.65 GiB), and the Figure 2 caption now gives both numbers.

We hope these additions address your concerns, and we would be glad to provide further results.

Dear Reviewer Juc6,

Thank you again for your thoughtful review and helpful comments on our paper.

We would like to briefly highlight the updates in our response. Following your comments, we compared ASLU with the one-sided update inside all three solvers, with every hyperparameter shared (§4.1 and Table 3; Tables A2 and A6). ASLU is clearly better when the correction is large or noisy (up to +4.0 dB), and the two are on par when it is small. This gap holds across three seeds (Table A8). We also compared against memory-saving techniques for decoder backpropagation, namely checkpointing and tiling (Table A1). In addition, we corrected the DM4CT memory claim against a decoder-only PSLD (77% reduction, Table A3) and added a Gauss–Newton view of the encoder update (§5). Finally, we credited the earlier suggestion of the one-sided update by Graikos et al. (NeurIPS 2025).

We sincerely appreciate the time and effort you have already devoted to reviewing our paper. If time permits before the discussion period closes, we would be grateful if you could read through our response. We would welcome any further thoughts and would be happy to clarify any remaining questions.

Best regards, The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Final Summary for the AC and Reviewers

We thank all three reviewers for their careful reviews. As the discussion period is getting to an end, below we summarize the strengths they noted, their shared concern and how we addressed it, and the other additions.

$ $

Strengths. The reviewers found ASLU simple and practical: two encoder forward passes replace decoder backpropagation, cutting peak memory by 64–85% at comparable quality across three solvers (Juc6, B9AV, M21t). They also found the drift/curvature analysis clear (Juc6, M21t), the experiments broad (B9AV, M21t), and the paper well written (Juc6, M21t).

$ $

Shared concern: does curvature cancellation matter beyond the one-sided update, which has the same cost? (Juc6 W1, B9AV W2, M21t W1.) Our new experiments give a two-part answer:

  • 1. When the correction is large or the measurement is noisy, ASLU is clearly better than the one-sided update.
  • 2. Otherwise, the two are on par, so using ASLU by default costs nothing.

The evidence is the new matched-Δx\Delta x experiment (§4.1, Table 3; 100 images, three seeds), in which both updates receive the same correction. For large corrections, ASLU wins on every metric by +1.0 to +4.0 dB; for small ones, including STeP and DM4CT (Tables A2, A6), the two are on par. The new matched-state analysis (App. C.3) explains this split: the one-sided error grows as O(α)O(\alpha) and ASLU's as O(α2)O(\alpha^2), as Eqs. (5) and (7) predict. Both Reviewer B9AV—who named this the main reason the work was not yet at acceptance—and Reviewer Juc6 raised their scores after our response.

$ $

Other concerns.

  • Why is the encoder update a sensible latent update? (Juc6 W2) We added a Gauss–Newton view to §5. When the encoder approximately inverts the decoder, ASLU's update approximates a Gauss–Newton step, while decoder backpropagation takes a plain gradient step.
  • Would standard memory-saving tricks close the gap? (Juc6 W3) We compared against gradient checkpointing and tiled decoding (Table A1). They narrow the gap, but decoder backpropagation matches ASLU's memory only with both combined, and it is then about 7× slower.
  • How exactly is PSLD modified? (B9AV W1) We now give the full PSLD+ASLU sampler (Algorithm 1) and an ablation of the gluing term (Table A5), and we corrected our earlier description of it.
  • Does ASLU help in every sampler? (B9AV Q3) Not always. In LATINO, which re-noises its latent at every step, simply re-encoding works better (App. C.7). We now state this scope clearly: ASLU is meant for samplers that carry the latent across steps.
  • How large are the corrections in practice? (M21t Q1) We measured them along the sampling trajectory (Table A9). The small and large settings differ by two to three orders of magnitude, which matches where ASLU's gain appears.

$ $

Thank you again for the suggestions, which have substantially strengthened the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.