- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes the Dynamics Hessian Penalty (DHP), which penalizes mixed latent derivatives of single-step conditional predictions in diffusion and rectified-flow models using stochastic finite differences. The goal is to encourage disentanglement while retaining the capacity for nonlinear composition through iterative generation.
Critical strengths
- Clear motivation and practical implementation. Appendix A explains why vanishing local mixed derivatives need not make the complete generative mapping additive; the authors also explicitly state that this does not provide an identifiability guarantee. The method avoids explicit Hessian computation and sampler unrolling during training, and adds no prediction branches at inference.
- Controlled evidence for incremental value. Table 3 uses the same ten training seeds and controls for BatchNorm. DHP-C improves mean DCI from 0.716 to 0.747 for noise prediction and from 0.686 to 0.726 for rectified flow. DHP-F shows improvements in the same direction. These results provide useful evidence that the regularizer contributes beyond normalization in the tested setting.
Critical weaknesses
- The novelty is incremental, and the relationship to prior work could be positioned more precisely. The authors should distinguish DHP more explicitly from the Hessian Penalty, COMET, DisDiff, and Decomp Diffusion, particularly by clarifying the relationship between DHP-C and the original HP estimator. The more compelling contribution is the choice of regularization target in per-step conditional training and its empirical effect, rather than independent novelty in the finite-difference estimator.
- Attribution of the largest gains remains incomplete. Some baselines in Tables 1 and 2 use settings from their original papers. Table 3 provides useful matched evidence, but does not fully explain the large gains on datasets such as Cars3D. A matched BN-only control on Cars3D would help isolate the contribution of DHP.
- CelebA provides limited evidence about cross-run stability. The authors clearly state that each model is represented by a single trained checkpoint and that the error bars come from repeated evaluation. These results therefore do not establish stability across independent training runs. Additional independent runs, or a more qualified robustness claim, would strengthen the real-image evidence.
Questions for authors
- Could the authors clarify the relationship between DHP-C and the original Hessian Penalty estimator, and focus the novelty claim on the regularization target and its empirical effects?
- Could a matched BN-only control be provided on Cars3D or another setting with large reported gains to better isolate DHP's contribution?
- Do the main CelebA findings persist across independently trained checkpoints? If additional runs are not feasible, could the limitation regarding training stability be stated more explicitly?
3 comments
Thank you for the detailed responses. The additional experiments and clarifications have addressed a lot of my concerns. However, some limitations remain (e.g., quantitative semantic controllability and its effect on generation quality). I will maintain my recommendation.
Thank you for taking the time to consider our responses and additional experiments. We appreciate your thoughtful review and constructive suggestions, which have helped us clarify the contribution and strengthen the evaluation.
Response to Reviewer 16Qn
Thank you for your thoughtful and encouraging review. We appreciate your positive assessment of DHP's motivation, practical implementation, and gains beyond BN1d in the controlled comparisons, as well as your constructive suggestions for strengthening the paper.
The relationship between DHP and HP
We agree that DHP’s main contribution lies in the regularization target and its empirical effects.
Section 4.7 (Table 7) now directly compares DHP-C fine-tuning (DHP-FT) with HP applied to the final latent of a five-step DDIM sampler (HP-FT), using the same initialization and 30,000 fine-tuning steps. DHP-FT achieves higher DCI on both datasets while reducing local Hessian energy and retaining 200-step sampling-map energy of the same order of magnitude as EncDiff w/BN1d. Appendix D.6 provides the corresponding Hessian visualizations.
We have clarified the comparison with COMET, DisDiff, and Decomp Diffusion in Section 2.2. COMET generates images by gradient descent on a sum of concept-conditioned energies. DisDiff adds factor-conditioned guidance fields to a pretrained unconditional diffusion model. Decomp Diffusion averages denoising predictions, each conditioned on a single component, during reconstruction and recombination.
At each generation step, the combined gradient or denoising prediction is additive across component codes, which may be multidimensional. With the current state held fixed, mixed derivatives between distinct component codes therefore vanish wherever they exist. DHP instead jointly conditions its denoising network on all latent coordinates and softly penalizes mixed derivatives between scalar coordinates.
Appendix B.3 now derives the relationship to the original HP estimator. DHP-C can be obtained from the original two-direction (k=2) HP construction by reparameterizing the sampled directions and accounting for normalization. We have also added an explicit citation to Peebles et al. (2020) in Section 3.3.
DHP-F uses a forward mixed-difference construction not provided in the original HP formulation. It reuses the base-loss prediction and requires three additional predictions, compared with four for DHP-C. Appendix B discusses this construction and its approximation-accuracy trade-off.
Matched BN-only control on Cars3D
Thank you for suggesting this control. We report the requested comparison below. All four methods use the same training budget, seed set, VQGAN, and evaluation data. Results are means and standard deviations over ten training runs with different seeds.
BN1d increases mean DCI by approximately 0.098. With BN1d held fixed, DHP-F and DHP-C provide additional mean DCI gains of approximately 0.115 and 0.143, respectively. FactorVAE is already close to its ceiling with BN1d, so its additional gains are small.
CelebA results across independent training runs
Following your suggestion, we evaluated five independently trained models for each DHP variant on CelebA. The following results are means and standard deviations across these five models.