acceptodds
Under review as a conference paper at ICLR 2027

DNPMorph: Direct Noise-Path Optimization for Smooth and Consistent Diffusion Morphing

Abstract

Text-to-image diffusion models possess vast knowledge of our visual world, however unlocking this prior for the task of image morphing remains a significant challenge. Here, we introduce DNPMorph, a diffusion-based morphing method that utilizes a pretrained diffusion model and, unlike existing methods, directly optimizes the noise latents of a morph path so as to minimize smoothness objectives on the corresponding diffusion-generated frames. To make path-level optimization practical, we use a type of zero-order optimization strategy which does not require differentiating through the diffusion sampling process, essentially treating it as a black box. The path is initialized by spherical linear interpolation between the initial noises corresponding to the user-provided endpoint images, and is then refined in a coarse-to-fine manner. We provide a theoretical conditional descent analysis for our method, and empirically compare it to existing diffusion-based morphing methods on four benchmarks. DNPMorph outperforms competing methods on all datasets in terms of path-smoothness metrics and on three of four in terms of FID, establishing a new state-of-the-art for diffusion-based image morphing.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
image morphing, diffusion models, image interpolation, generative modeling

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_SrMuOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces DNPMorph, a novel approach for image morphing using pretrained latent diffusion models. Unlike existing methods that indirectly improve morphing smoothness by interpolating model parameters, modifying attention mechanisms, or constructing complex geometric priors, DNPMorph formulates the task as the direct optimization of the noise-path latents to minimize a sequence-level smoothness objective evaluated on the decoded frames. To make this path-level optimization computationally feasible, the authors employ a zero-order surrogate update that treats the diffusion denoising chain as a black box (omitting backpropagation through it) while still propagating gradients through the decoder and the image-space smoothness loss. The path is initialized via Spherical Linear Interpolation (SLERP) in the native Gaussian noise space and refined using a two-stage coarse-to-fine schedule. The paper provides a theoretical conditional descent analysis to justify the surrogate update and conducts extensive experiments across four benchmarks. DNPMorph achieves state-of-the-art performance in path-smoothness metrics and improves FID on three out of four datasets, all while preserving the native Gaussian geometry of the diffusion prior.

Critical strengths
  1. Elegant and Direct Problem Formulation: The core idea of directly optimizing the noise latents to minimize decoded-image smoothness is highly intuitive. By keeping the diffusion model fixed and treating morphing as a trajectory optimization problem in the native noise space, the method avoids the pitfalls of adversarial exploitation seen in direct pixel-space optimization (as shown in Sec 5.3), while ensuring that all intermediate frames remain strictly on the diffusion model's data manifold.

  2. Comprehensive and Rigorous Validation: The paper provides an exceptionally thorough evaluation. Beyond standard quantitative metrics and user studies, the authors conduct insightful analyses to prove that the optimized latents do not arbitrarily drift but actually preserve the Gaussianity and the geometric arc of the initial SLERP path (Section 5.4). Furthermore, the inclusion of a theoretical conditional descent analysis (Theorem 1) for the zero-order surrogate update adds significant mathematical rigor to the proposed optimization strategy.

Critical weaknesses
  1. High Test-Time Computational Cost: As acknowledged in the limitations and runtime analysis, DNPMorph requires iterative test-time optimization (invoking the full diffusion-process map multiple times per sequence), making it significantly slower than methods that rely on single-pass sampling dynamics or parameter interpolation (e.g., FreeMorph or GeoMorph). While the authors reasonably frame this as a "quality-compute tradeoff," this latency restricts its applicability in interactive, real-time, or large-scale batch-processing scenarios.

  2. Lack of Explicit Structural/Semantic Correspondence: The method relies entirely on local, adjacent-frame smoothness objectives (like LPIPS and DINO distances) without explicit long-range correspondence, optical flow, or 3D-aware motion modeling. Consequently, when morphing between endpoint images with drastic structural, viewpoint, or semantic changes, the optimization might struggle to produce physically plausible intermediate motions, potentially leading to "melting" artifacts or semantic confusion (as noted in the limitations).

Questions for authors
  1. Could the authors discuss potential avenues for accelerating this test-time optimization in future work or the rebuttal? For instance, would leveraging consistency distillation, few-step flow models, or initializing the path with a faster, lightweight proxy model reduce the required number of optimization iterations without sacrificing the final smoothness?

  2. While this is a common limitation in diffusion morphing, how sensitive is the method to extreme viewpoint changes compared to structural baselines? Incorporating a lightweight structural prior (e.g., depth-aware warping or optical flow guidance) as an additional regularization term in the sequence-level objective could further enhance the method's robustness. Have the authors considered or experimented with any structural regularizers?

3 comments

We thank the reviewer for the constructive review.

Test-time cost (W1): The revised Table 1 reports runtime and peak memory for all methods. The peak memory of DNPMorph (about 36 GB) is similar to that of IMPUS and higher than that of the single-shot baselines (5-9 GB). Two observations put this cost in context. First, at a comparable budget DNPMorph is clearly better: DNP-50 and IMPUS require similar time (20 vs. 17 minutes per pair) and similar memory, and DNP-50 outperforms IMPUS on every metric and every dataset. Second, the faster baselines are single-shot methods and cannot exchange additional compute for smoother morphs. DNPMorph, in contrast, improves gradually along the optimization, so the user can choose an operating point. We position DNPMorph for offline, high-quality morph generation and state this explicitly in the main text.

Accelerating the optimization (Q1): We thank the reviewer for these suggestions, which fit naturally into our framework. Each iteration requires only forward passes through the TT-step sampler and no backward pass through the denoiser.

Consistency models [1], latent consistency models [2] and adversarially distilled models [3] reduce the number of steps from 50 to just a few. Since our method only requires a deterministic mapping from noise to clean latents, they can be used directly. Exact verification of the alignment condition also becomes cheaper in this regime.

A second avenue is initialization. Since DNPMorph locally refines its initial path (Sec. 5.4), starting from a path that is already smooth, e.g., one produced by a faster morphing method, should reduce the required number of iterations. Finally, most of the improvement occurs early in the dense stage (Fig. S14), so an early-stopping rule offers a simple further saving. We will add this discussion to the paper and make an effort to include a few-step variant in the final version.

Structural correspondence and extreme viewpoint changes (W2/Q2): We agree that explicit structural correspondence could be particularly valuable for handling large viewpoint and pose changes. Our current LPIPS and DINO objectives encourage perceptual and semantic smoothness between adjacent frames, but do not explicitly enforce physically plausible motion. Consequently, challenging transitions can still exhibit abrupt pose changes or cross-fade-like effects, as illustrated by the failure cases in App. G. Similar artifacts also appear in competing methods (Figs. 5 and S10-S12), although we have not conducted a separate quantitative comparison specifically for extreme viewpoint changes.

We have considered incorporating structural guidance into the objective, as the reviewer suggests, and view this as a particularly promising direction for future work. A key advantage of our direct optimization formulation is that the sequence-level objective S\mathcal S can naturally accommodate additional terms, such as optical-flow-based warp consistency [4] or depth-aware correspondence, without modifying the optimization framework. Such terms could encourage more coherent motion and structural consistency, complementing the current perceptual smoothness objectives. We have added this discussion to Sec. 5.5 and App. G of the revised manuscript.

[1] Y. Song, P. Dhariwal, M. Chen, and I. Sutskever. Consistency models. ICML, 2023.

[2] S. Luo, Y. Tan, L. Huang, J. Li, and H. Zhao. Latent consistency models: Synthesizing high-resolution images with few-step inference. arXiv, 2023.

[3] A. Sauer, D. Lorenz, A. Blattmann, and R. Rombach. Adversarial diffusion distillation. ECCV, 2024.

[4] Z. Teed and J. Deng. RAFT: Recurrent all-pairs field transforms for optical flow. ECCV, 2020.

Thanks for the rebuttal. I will maintain my 3: Weak acceptance for the paper.

We thank the reviewer for their constructive and positive feedback. Their review helped us improve the manuscript.

Reviewer_Txj5OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

DNPMorph frames diffusion morphing as direct optimization of the interior noise-path latents of a fixed diffusion model, minimizing a decoded-sequence smoothness objective (LPIPS-PPL, DINO-PPL, PDV). To avoid differentiating through the denoising chain, it uses a zero-order surrogate: stop-gradient through the diffusion map, backprop only through the decoder and the path loss. The path is initialized by SLERP between the inverted endpoint noises and refined coarse-to-fine (sparse frames {4,8,12}, then all interior frames). Theorem 1 gives a conditional sufficient-decrease guarantee under an alignment condition between the surrogate and true gradients. Experiments run on MorphBench, Morph4Data, and author-built AFHQ/CAHQ subsets against SLERP, DiffMorpher, IMPUS, FreeMorph, and GeoMorph, all under a single SD 2.1 backbone.

Critical strengths

The empirical controls are genuinely good, and they are the strongest part of the paper. The direct-Y-optimization ablation (Table S4) shows that optimizing decoded frames directly drives the training losses even lower than DNPMorph but wrecks held-out DreamSim-PPL (0.50 vs 0.25) and FID (80 vs 52); and DreamSim-PPL, which is not in the objective, ends up best on all four datasets. Together this is real evidence that the noise-path parameterization prevents adversarial exploitation of the feature losses rather than just fitting them. The statistical rigor (paired bootstrap CIs, a 70-participant user study with Wilson CIs) is also above the norm for this area.

Critical weaknesses
  1. The core mechanism is not new. The paper's own related work concedes that FlowOpt already optimizes latents through a full denoising/flow process without backpropagating through the model, and that Jacobian-free generative optimization (SDS/DDS/PDS) is common. DNPMorph's surrogate — stop-gradient through the chain, backprop through the decoder — is essentially that same scheme applied to a multi-frame path with an off-the-shelf smoothness loss. Approximating the chain Jacobian by the identity is about as simple as a zero-order surrogate gets. The delta is a new application (morphing) plus a sequence-level objective, not a new optimization idea.

  2. The theory does not underwrite the method where it is actually used. Theorem 1 is a conditional descent lemma whose key alignment assumption is probed only at T=10 with lambda=0, and the authors explicitly state they cannot show it holds at the T=50 operating point. So for the reported configuration the guarantee is largely decorative.

  3. Narrow scope and unquantified-in-the-main-text cost. Everything is SD 2.1; there is no evidence the alignment condition transfers to SDXL/FLUX/SD3, which is exactly where it is most in doubt. Test-time cost is high (each iteration runs the full 50-step map for all interior frames) and is quantified only in Appendix F (Table S8: 20/45 min per pair for DNPMorph-50/-150 vs. 0.75-17 min for baselines), while the main-text tables never pair the quality numbers with these runtimes — so the "new SOTA" claim is a 25-60x slower operating point than the fastest baselines, which is easy to miss. It also does not win FID everywhere (Morph4Data: 103.4 vs FreeMorph 93.1, GeoMorph 93.9).

Questions for authors
  1. Beyond the change of task and objective, what is the technical difference between your surrogate update and FlowOpt's? Is anything in the update specific to path-level morphing, or would FlowOpt's optimizer applied with a smoothness loss reproduce it?

  2. Can you verify the alignment condition at the actual T=50, lambda=0 operating point, or state plainly that Theorem 1 does not apply to the reported configuration?

  3. Table S8 reports 20/45 min per pair for DNPMorph-50/-150 against 0.75-17 min for the baselines, yet these runtimes never appear alongside the main results. Why were they omitted from the main comparison, and how does the smoothness advantage look at a compute budget matched to GeoMorph or DiffMorpher (i.e., points on the Fig. S14 trajectory, not just the end states)?

2 comments

We thank the reviewer for the detailed review.

Technical difference from FlowOpt (W1/Q1): We agree that the underlying optimization principle is closely related to FlowOpt: both avoid differentiating through the diffusion process by replacing its Jacobian with the identity. We do not claim a new optimizer, and have revised the manuscript to explicitly describe our update as a straight-through-type surrogate [1]. Our contributions instead concern the morphing formulation and its optimization:

  1. Structured path optimization: Unlike FlowOpt's single-image target optimization, DNPMorph jointly optimizes the interior noise latents of a morph path with fixed endpoints. The sequence-level loss couples neighboring frames, and we use a coarse-to-fine schedule to improve the resulting path. The single-stage ablation (Tab. S1) confirms the benefit of this schedule, particularly for FID (52.2 vs. 61.2).

  2. The role of the diffusion prior: Our direct-Y\boldsymbol Y control (Tab. S4) optimizes the same objective directly over decoded frames. Although it achieves substantially lower optimized losses, it barely improves the visual morph (Fig. S3), while performing considerably worse on held-out DreamSim-PPL and FID. This demonstrates how unconstrained frame optimization can exploit the feature losses, whereas DNPMorph produces meaningful visual improvements that generalize to held-out metrics. Thus, optimizing the noise path through the diffusion model, rather than the frames themselves, is essential to the observed morphing improvements.

  3. Analysis for sequence-level objectives: FlowOpt's convergence analysis relies on a contraction argument for target-matching objectives. Our morphing objective has no prescribed target sequence or known minimum. We therefore provide a conditional descent analysis based on the alignment between the surrogate and true gradients, and empirically verify this condition at the reported configuration (W2/Q2).

Accordingly, the revised contribution list emphasizes our morphing formulation, coarse-to-fine refinement, and analysis, without claiming novelty for the underlying Jacobian-free optimization principle.

Does Theorem 1 apply at T=50T{=}50, λ=0\lambda{=}0 (W2/Q2): The submission verified the condition of Theorem 1 only at T=10T{=}10. In the revision, we verify it at the reported configuration. Specifically, we computed the exact gradient ∇Ψ\nabla\Psi by backpropagating through the full 50-step DDIM chain, using per-step checkpointing, and compared it with the surrogate direction qq used by the optimizer. We evaluated the alignment constant γ\gamma along the actual optimization trajectories of 20 MorphBench pairs, in both stages, from the SLERP initialization to the final iterate. We also evaluated it on random Gaussian paths following the protocol of App. B. In both settings, γ\gamma was positive at every evaluated point, and the smoothness and gradient-norm conditions behaved as they did at T=10T{=}10. The alignment condition of Theorem 1 therefore holds throughout the configuration we report. In the revised manuscript we moved these results into the main text next to the theorem.

We also explain why γ\gamma should be positive. Since q=gq=g and ∇Ψ=JF⊤g\nabla\Psi=J_F^\top g, the inner product ⟨∇Ψ,q⟩=g⊤JF g\langle\nabla\Psi,q\rangle=g^\top J_F\,g is a quadratic form of the diffusion-map Jacobian. It is therefore positive whenever JFJ_F is close to symmetric positive definite, as expected if the DDIM map is close to an optimal-transport map [4].

Scope, runtime and FID on Morph4Data (W3): Other backbones - DNPMorph applies to any diffusion model with a deterministic mapping from noise to clean latents. Following the reviewer's comment, we applied it to SDXL [5] and compared DNPMorph-50 with its SDXL SLERP initialization on CAHQ and AFHQ:

DatasetMethodFID ↓LPIPS-PPL ↓PDV ↓DINO-PPL ↓DreamSim-PPL ↓
CAHQSLERP60.31.320.051.300.42
CAHQDNPMorph-5053.81.040.030.930.26
AFHQSLERP47.61.840.071.610.50
AFHQDNPMorph-5041.81.410.040.970.23

DNPMorph improves all five metrics on both datasets, including the held-out DreamSim-PPL. We have added these results to App. E of the revised manuscript, together with qualitative SDXL examples. We chose SD 2.1 for the main comparison because many prior diffusion-morphing methods were developed on Stable Diffusion backbones of this family. Running all methods on the same backbone ensures that the comparison reflects the morphing method rather than the generator. On this backbone DNPMorph already achieves satisfying results, and the SDXL experiment shows that they are not specific to it.

Runtime (Q3): The revised Table 1 reports runtime and peak memory for all methods. A comparison at a similar budget already exists: IMPUS and DNP-50 require comparable time (17 vs. 20 minutes per pair) and comparable peak memory (about 35 GB), and DNP-50 is better on every metric and every dataset. The faster baselines, GeoMorph and DiffMorpher, are single-shot methods, and at their budgets the current implementation is not competitive (the sparse stage alone already exceeds them), and we clarified it in the abstract that our state-of-the-art claim refers to a higher compute budget. Note that the cost is not fixed: each iteration requires only forward passes through the denoiser, so it scales with the number of sampling steps. Few-step models [6, 7] would therefore translate directly into faster optimization.

FID on Morph4Data: On Morph4Data, DNPMorph improves FID over SLERP, DiffMorpher and IMPUS, while FreeMorph and GeoMorph attain lower FID; at the same time, DNPMorph obtains the best path-length metrics. Many pairs in this dataset involve large semantic gaps between the endpoints, and we hypothesize that on such pairs a smoother path may pass through intermediate content that is further from both endpoint distributions. We have added this discussion to Sec. 5.3 in the revised manuscript.

[1] Y. Bengio, N. Léonard, and A. Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv, 2013.

[2] D. P. Bertsekas and J. N. Tsitsiklis. Gradient convergence in gradient methods with errors. SIAM J. Optim., 2000.

[3] A. Ajalloeian and S. U. Stich. On the convergence of SGD with biased gradients. arXiv, 2020.

[4] V. Khrulkov, G. Ryzhakov, A. Chertkov, and I. Oseledets. Understanding DDPM latent codes through optimal transport. ICLR, 2023.

[5] D. Podell et al. SDXL: Improving latent diffusion models for high-resolution image synthesis. ICLR, 2024.

[6] Y. Song, P. Dhariwal, M. Chen, and I. Sutskever. Consistency models. ICML, 2023.

[7] A. Sauer, D. Lorenz, A. Blattmann, and R. Rombach. Adversarial diffusion distillation. ECCV, 2024.

Reviewer_9RHUOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes a method for image morphing given two endpoint images and a pretrained diffusion model, where the objective is to generate a sequence of images that connect the two endpoints such that the intermediate images have smooth transitions and each image is relatively plausible (the latter may be somewhat undefined, depending on the endpoint images). To this, I would add directness: a smooth, plausible path that meanders should be preferred less than a shorter, smooth, plausible path. The approach is to optimize a path of noise latents with respect to an image-sequence-space objective (perceptual and semantic smoothness terms), starting from a SLERP path. This has some applications in animation and temporal video upsampling. The claimed contributions are (a) the method; (b) the optimization strategy; and (c) identifying the conditions for objective decrease. The approach is evaluated on four benchmarks using Stable Diffusion 2.1.

Critical strengths
  1. The main idea of directly optimizing the metrics that you are to be assessed on is sensible, with Goodhart's Law as a caveat: "When a measure becomes a target, it ceases to be a good measure." For the latter, the authors do consider metric transfer to a limited extent and compare with directly optimizing the images themselves, showing that the prior of the diffusion model helps in this regard. This was warranted, and strengthened the paper.

  2. The coarse-to-fine design is sensible, and their approach really does lend itself to this strategy; it works well.

  3. I appreciate that the authors have considered the consequences of the zero-order approximation on the optimization efficiency.

  4. The evaluation metrics are comprehensive and sufficient, and evaluating on four datasets is a decent amount for this task.

Critical weaknesses
  1. Runtime and memory. 45 minutes per image pair (reported in Appendix F) is exceedingly computationally expensive for an inference-time method, and is an order of magnitude more than the compared methods. Geomorph and diffmorpher are also pareto optimal w.r.t. the proposed method on the runtime vs performance plots of Fig S14 of the appendix. Moreover, the peak memory usage, while not reported, is likely to be significantly higher than the other methods. Runtime and peak memory must be reported in Table 1, since they are going to be primary considerations for any user of the method.

  2. Metric hacking. Notwithstanding my earlier comments, directly optimizing the semantic and perceptual path lengths makes improvements on these metrics entirely unsurprising. Given that these metrics are surrogates for some slightly vaguer notion of a "good morph", the paper does not provide valuable take-away messages for the community. In contrast, some of the existing work approach the problem from the other direction: given this idea of a good morph, what might that look like in diffusion latent space? What is its geometry, and what properties might it have? While indirect, I would expect to learn more from such an approach.

  3. Contributions.

    1. Theorem 1 is an existing result in the optimization literature (it appears, for example, in proofs about stochastic gradients and momentum gradients). Which is not to say that it's not useful to include, since it makes it clear why the approximation leads to inefficient optimization for small gamma, but it is a stretch to frame it as a contribution.
    2. The optimization strategy, where a pass-through gradient is used for the diffusion model, is listed as a contribution, but the authors are clearly aware from their related work section that this is an existing technique. This limits the contribution to an application of a known technique to the image morphing problem.
  4. Given that this paper is about direct optimization of an objective function, including said objective function in the main paper is rather important. If space is the limiting factor, I'd recommend moving section 5.4 to the appendix: confirming Gaussianity and small deviation from the SLERP paths does not seem particularly important.

  5. Assumption. The alignment condition (eq 9) is the big assumption, and it is not really clear why it should be true in high dimensions, where almost all vectors are orthogonal. This makes the empirical evidence for positive gamma to be rather important, and would be better located in the main paper than the appendix.

  6. Evaluation.

    1. The method is evaluated only on Stable Diffusion 2.1, which is significantly outdated now. Evaluation on more modern open-source models, such as the FLUX families, would seem necessary.
    2. Result quality. Despite optimizing an objective that should penalize this, there are still many abrupt transitions in the results, such as in Fig 4 (middle right), and the same sequence in Fig 5, which largely correlate with pose changes. Surely a direct image-sequence-space objective would have been the perfect place to provide a signal to address this long-standing weakness in morphing approaches.
  7. Clarity.

    1. Fig 2 appears to indicate that the method has no inputs (the endpoint images to noise mapping is missing) and that the entire set of noise latents is optimized, instead of z_1 to z_N-2.
    2. Since the main experiments sets λ=0\lambda=0, the regularization term should not be presented as part of the method: it is misleading. There are conceptual issues with lambda>0 in any case, since the resulting path could not be optimal with respect to the metrics if so regularized.
Questions for authors
  1. Do the authors have a hypothesis for why gamma is positive?

  2. Could the direct optimization approach be used to reduce abrupt pose transitions?

6 comments

We thank the reviewer for the thorough and constructive review.

Runtime and memory (W1): Table 1 now reports runtime and peak GPU memory for all methods. The peak memory of DNPMorph is about 36 GB. This is on par with IMPUS (35 GB) and higher than the single-shot baselines FreeMorph, GeoMorph and DiffMorpher (5-9 GB). It remains bounded because we never backpropagate through the denoising chain. The UNet runs only in the forward direction, and gradients flow only through the VAE decoder and the perceptual networks.

Regarding Fig. S14, note that FreeMorph, GeoMorph and DiffMorpher are single-shot methods. Each occupies a single point on the runtime-quality plane and cannot exchange additional compute for a smoother morph. DNPMorph, in contrast, traces a curve along which the user can choose an operating point. At a comparable budget, DNP-50 outperforms IMPUS (20 vs. 17 minutes per pair, with similar peak memory) on all metrics and all datasets. We do not claim that DNPMorph is fast, and as we state in our paper, we target an offline, high-quality morphing.

Metric hacking and take-away (W2): We agree that improvements on the optimized metrics alone would not be convincing, which is why we also evaluate generalization to held-out metrics. DreamSim-PPL, which is not part of the objective, improves on all four datasets, and FID improves on three of four. Moreover, App. C.6 shows that held-out DreamSim-PPL decreases alongside the optimized metrics throughout optimization (Fig. S4), while DreamSim-PDV is best on three of four datasets (Tab. S5). The user study further supports the perceptual quality of our morphs.

Most importantly, the direct-Y\boldsymbol Y control (Tab. S4) optimizes the same objective directly over decoded frames. Despite achieving substantially lower optimized losses than DNPMorph, it performs considerably worse on held-out DreamSim-PPL and FID. As illustrated in Fig. S3, direct-Y\boldsymbol Y optimization exploits the optimized metrics without meaningfully improving the visual transition, whereas DNPMorph produces a substantially smoother morph. This demonstrates metric exploitation by direct frame optimization, while DNPMorph's improvements transfer to held-out metrics and visual quality, providing evidence against the same metric-hacking behavior.

This control also suggests the take-away the reviewer asks for: the diffusion map itself acts as the morphing prior. Once the frames are constrained to be outputs of the generator, losses that are easily exploited in pixel space become a meaningful signal. Sec. 5.4 further indicates where good morphs live in latent space. The optimized latents remain Gaussian and stay close to the SLERP arc, deviating from it by a moderate correction.

Contributions (W3):

(1) Theorem 1 can be seen as reminiscent of descent analyses of gradient methods with inexact or biased directions [1, 2], and we now state this explicitly after the theorem. The difference lies in the setting: we analyze the update within zero-order optimization through the entire diffusion process, where the inexactness stems from omitting the Jacobian of the denoising chain. This reduces the validity of the surrogate to a single condition, Eq. (9), that can be measured on a real model. In the revision we measure it at the actual operating point (see W5 below).

(2) The surrogate is indeed a straight-through-type estimator [3], and we revised the manuscript accordingly. The contribution we claim is the formulation, in which morphing becomes optimization of the noise path through a fixed diffusion model under a sequence-level loss on the decoded frames. We also show that this parameterization, rather than the loss alone, is what makes the optimization meaningful (W2). The contribution list is revised accordingly.

Objective in the main paper (W4): We agree with the reviewer that the explicit optimization objective is important to understanding the method, and appreciate this suggestion. Accordingly, we have moved the instantiated loss objective into Sec. 4.2 of the revised manuscript.

The alignment condition, and why γ>0\gamma>0 (W5/Q1): We thank the reviewer for highlighting this central assumption. In direct response, we conducted a new experiment to verify the alignment condition at the actual experimental configuration (T=50T{=}50, λ=0\lambda{=}0), extending our previous diagnostics at T=10T{=}10. Specifically, we computed the exact gradient ∇Ψ\nabla\Psi by backpropagating through the full 50-step DDIM chain, using per-step checkpointing, and compared it with the surrogate direction qq used by the optimizer. We evaluated γ\gamma along the real optimization trajectories of 20 MorphBench pairs, from the SLERP initialization to the final iterate, and on random Gaussian paths following the protocol of App. B. In both settings, γ\gamma was positive at every evaluated point. Interestingly, γ\gamma grows during optimization: as the path becomes smoother, the surrogate becomes better aligned with the true gradient. The smoothness and gradient-norm conditions behave as they did at T=10T{=}10. These results now appear next to Theorem 1 in the main text.

The positivity of γ\gamma may seem surprising, given that random high-dimensional vectors are nearly orthogonal. Write g=JD⊤∇YSg=J_{\mathcal D}^\top\nabla_{\boldsymbol Y}\mathcal S. Then q=gq=g and ∇Ψ=JF⊤g\nabla\Psi=J_F^\top g, so that ⟨∇Ψ,q⟩=g⊤JF g\langle\nabla\Psi,q\rangle=g^\top J_F\,g. This is a quadratic form of the Jacobian of the diffusion map, not an inner product between two unrelated vectors. It is therefore positive whenever the symmetric part of JFJ_F is positive definite along gg. Indeed, we measure a substantial cosine similarity between ∇Ψ\nabla\Psi and qq, far from the near-zero value expected for unrelated vectors. One possible explanation is that the deterministic DDIM map is close to an optimal-transport map [4]. By Brenier's theorem [5], such a map is the gradient of a convex potential, and its Jacobian is thus symmetric positive semi-definite. We add this discussion to the paper as a hypothesis that is consistent with our measurements.

Evaluation on SD 2.1 only (W6 (1)): DNPMorph is not tied to SD 2.1. It applies to any latent diffusion model with a deterministic mapping from noise to clean latents. Following the reviewer's comment, we applied it to SDXL [6] and compared DNPMorph-50 with its SDXL SLERP initialization on CAHQ and AFHQ:

DatasetMethodFID ↓LPIPS-PPL ↓PDV ↓DINO-PPL ↓DreamSim-PPL ↓
CAHQSLERP60.31.320.051.300.42
CAHQDNPMorph-5053.81.040.030.930.26
AFHQSLERP47.61.840.071.610.50
AFHQDNPMorph-5041.81.410.040.970.23

On both datasets, DNPMorph improves all five metrics, including FID and the held-out DreamSim-PPL. We have added these quantitative results, together with qualitative SDXL examples, to App. E of the revised manuscript.

We chose SD 2.1 for the main comparison deliberately. Many prior diffusion-morphing methods were developed on Stable Diffusion backbones of this family, and running all methods on the same backbone ensures that the comparison reflects the morphing method rather than the generator. On this backbone DNPMorph already achieves satisfying results, and the SDXL experiment shows that these results are not specific to it. We will make an effort to include FLUX in the final version.

Abrupt pose transitions (W6 (2)/Q2): Our objective sums adjacent-frame LPIPS and DINO distances, which measure how much the image changes between frames but not how it changes. A gradual rotation and a quick cross-fade between two poses can therefore incur similar per-frame costs. Moreover, the optimization refines the SLERP path locally, whereas a physically smooth pose change often lies far from it.

The direct formulation is, in fact, the natural place to address this. The objective S\mathcal S is modular, so a flow-based warp-consistency term [7] or a depth-aware term can be added without changing the optimizer. We have added this discussion to the limitations and regarded it as a promising extension of the method.

Clarity (W7):

(1) We clarified the caption of Figure 2: it now states that the endpoint latents are obtained by inverting the input images and remain fixed, and that only z1,…,zN−2\boldsymbol z_1,\dots,\boldsymbol z_{N-2} are optimized.

(2) We agree that the main method uses λ=0\lambda=0, however we intended to present it in the most the general case. We have revised the experiments section accordingly and emphasized it. For λ>0\lambda>0, the solution trades optimality with respect to the smoothness metrics for proximity to the reference path.

[1] D. P. Bertsekas and J. N. Tsitsiklis. Gradient convergence in gradient methods with errors. SIAM J. Optim., 2000.

[2] A. Ajalloeian and S. U. Stich. On the convergence of SGD with biased gradients. arXiv, 2020.

[3] Y. Bengio, N. Léonard, and A. Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv, 2013.

[4] V. Khrulkov, G. Ryzhakov, A. Chertkov, and I. Oseledets. Understanding DDPM latent codes through optimal transport. ICLR, 2023.

[5] Y. Brenier. Polar factorization and monotone rearrangement of vector-valued functions. Comm. Pure Appl. Math., 1991.

[6] D. Podell et al. SDXL: Improving latent diffusion models for high-resolution image synthesis. ICLR, 2024.

[7] Z. Teed and J. Deng. RAFT: Recurrent all-pairs field transforms for optical flow. ECCV, 2020.

Comments on the paper

1 comment

We thank the reviewers for their constructive feedback. We greatly appreciate their suggestions which we believe resolve their concerns and have helped us strengthen the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.