From Decoder Backward to Encoder Forward: Antisymmetric Latent Updates for Diffusion Inverse Solvers

High-dimensional and high-resolution inverse problems are difficult to solve with latent diffusion inverse solvers (DIS) because measurement consistency is imposed in pixel space, while the generative prior evolves in latent space. Thus, every sampling step must convert a pixel-space correction $\Delta x$ into a latent update $\Delta z$. Most existing methods perform this conversion by backpropagating through the decoder, which dominates runtime and memory at high resolution; forward-only alternatives such as direct re-encoding avoid this cost but distort the latent prior enough to degrade reconstruction quality. We propose Antisymmetric Latent Update (ASLU), $\Delta z=\rho \cdot \frac{1}{2}[\mathcal{E}(x+\alpha \Delta x)-\mathcal{E}(x-\alpha \Delta x)]$: an antisymmetric difference of two encoder forward passes that vanishes at $\Delta x=0$, preventing accumulated drift, and cancels second-order encoder-curvature bias while preserving the intended first-order Jacobian response. Because ASLU consumes only $\Delta x$—never a gradient of the decoder or the measurement operator—the same rule applies whatever DIS mechanism produces the correction. Controlled experiments show that curvature cancellation matters increasingly for larger corrections, while drop-in use in high-resolution image inverse problems and high-dimensional black-hole video reconstruction reduces peak memory by up to $64\%$ and $76\%$, respectively. On an independent sparse-view CT benchmark against multiple latent DIS methods, ASLU maintains competitive reconstruction quality in a low-memory, short-runtime region not reached by the evaluated baselines.