acceptodds
Under review as a conference paper at ICLR 2027

DOHF: Online Diffusion Fine-tuning with Doob's -transform Guidance

Abstract

Reward-based diffusion fine-tuning faces practical challenges when desirable outcomes are rare or conditioning corrections are costly to estimate. In this work, we propose Diffusion Online -guidance Fine-tuning (DOHF), which turns Doob's -transform into a practical online training algorithm. DOHF assigns optimality weights to generated samples, estimates the normalized local correction under the current rollout policy, and distills it directly into the generative model. Theoretically, we characterize the population-optimal DiffusionNFT update as well as the various classfier free guidance methods through a unified -transform perspective. Methodologically, our framework accommodates black-box and non-differentiable rewards without additional network evaluations. We further show improved alignments under three empirical scenarios. Our work demonstrates how adapting probabilistic conditioning through inexpensive estimation and iterative distillation can improve generative learning across statistical sampling and visual generation.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
diffusion models, post training, Doob's h-transform

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_AXACOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes DOHF, an online diffusion fine-tuning framework based on Doob's h-transform. It expresses a terminal reward-weighted distribution through a local score or velocity correction, estimates this correction using sampled transitions and reward evaluations, and distills it into the model through regression. The paper also characterizes the population-level relationship to DiffusionNFT, identifying a state-dependent scaling that recovers the ideal correction. Experiments cover Gaussian rare-mode adaptation, constrained contingency-table generation, and SD3.5-Medium text-to-image fine-tuning, with improvements over the base model and several adaptation baselines.

Critical strengths
  1. Useful theoretical interpretation. The connection between terminal distribution tilting, local velocity correction, and DiffusionNFT's population optimum is informative. Under the idealized assumptions, the derivation appears sound and explains the role of state-dependent guidance strength. Its main value is a clearer interpretation of existing update mechanisms, which I consider a meaningful contribution.

  2. Practical realization with supporting evidence. Directly regressing the estimated correction provides a concrete fine-tuning procedure compatible with non-differentiable rewards, without training a separate critic. The statistical and text-to-image experiments demonstrate practical utility beyond the theoretical interpretation and support a positive assessment.

Critical weaknesses
  1. The connection between the ideal theory and the practical reward estimator needs stronger justification. The theoretical correction depends on h(x_t,t,c) = E[w(X_0) | X_t=x_t,c], whereas Section 3.1 replaces complete continuations with a one-step terminal prediction and evaluates the terminal reward on that prediction. A predicted clean image need not receive a score representative of its eventual completion, especially at high noise levels. This affects the reward weights and consequently the estimated correction, rather than merely adding sampling variance; increasing the number of candidates alone does not remove the surrogate bias. Relevant prior work, SPO, adapts PickScore to timestep-dependent denoised predictions (Section 5.1, https://arxiv.org/html/2406.04314v1), reinforcing the need to validate this assumption. This does not invalidate the h-transform identity or establish that DOHF fails, but it limits what that identity guarantees for the implemented algorithm. The paper should distinguish its exact population-level characterization from its approximate estimator and finite-step training procedure, with evidence that the surrogate preserves useful reward ordering or correction directions across timesteps.

  2. A controlled ablation of the state-dependent normalization is missing. Section 3.3 emphasizes the distinction between the normalized correction, grad log h, and the h-scaled correction associated with DiffusionNFT, arguing that normalization avoids attenuating updates in low-h regions. However, the method-level comparisons do not isolate this mechanism. A useful ablation would compare the normalized and unnormalized estimators within the same regression framework, using the same rollout protocol, reward estimator, and training budget, while tuning a global correction scale for each variant. This would distinguish the value of state-dependent scaling from a change in overall update magnitude. An analysis of progress and estimator variability in low-h regions would further test the specific mechanism proposed by the theory. This matters because recovering the ideal correction does not by itself establish an empirical advantage under finite-sample estimation.

Questions for authors
  1. Which exactness claims remain valid after introducing the one-step surrogate, finite-sample normalization, and approximate regression? What additional assumptions are required?

  2. Can a small diagnostic compare one-step predictions with complete continuations across noise levels, measuring reward-ranking agreement or agreement between the resulting correction directions?

  3. Can the authors isolate state-dependent normalization through a controlled ablation against an unnormalized correction with a tuned global scale, and report whether its benefit is concentrated in low-h regions as suggested by Section 3.3?

Reviewer_hHLKOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes DOHF, a method for fine-tuning diffusion and flow models using Doob’s h-transform. It estimates normalized reward guidance under the current generator and distills the resulting correction into the model through velocity regression. A one-step prediction of the final output is used to reduce estimation cost. The paper relates the proposed update to DiffusionNFT and evaluates it on Gaussian mixture adaptation, contingency-table generation, and text-to-image fine-tuning.

Critical strengths

The h-transform formulation provides a useful interpretation of reward-based fine-tuning, and the connection to DiffusionNFT clarifies the role of state-dependent guidance strength. The experiments cover both statistical sampling and image generation, with positive results for rare-mode adaptation and improvements on several image reward metrics.

Critical weaknesses
  1. The main GenEval comparison does not establish an advantage under comparable inference computation. In Table 3, DOHF without CFG achieves 44.27%, below DiffusionNFT’s 49.39%. Its stronger result of 62.52% uses CFG, which requires both conditional and unconditional predictions at each sampling step. Although unguided DOHF improves several other metrics, the headline GenEval advantage combines the fine-tuning method with additional inference-time guidance. Moreover, the claim that CFG is unavailable to DiffusionNFT requires clarification: the original NFT paper also reports using CFG in Appendix D, Table 2. The available comparisons therefore leave DOHF’s accuracy–compute advantage over NFT unclear.

  2. The practical estimator and the benefit of normalization are insufficiently validated. The theoretical analysis concerns the exact h-transformed target, but Section 3.1 replaces completed trajectories with one-step predictions of their final outputs. The paper does not show how accurately this shortcut estimates the required correction, particularly at noisy timesteps. In addition, while the population analysis highlights normalization by (h) as the key distinction from NFT, DOHF also changes the guidance estimator and training objective. The comparison between the complete methods therefore does not isolate the normalization’s contribution. A controlled comparison within DOHF, with and without the (h)-dependent normalization while keeping the other components fixed, is missing. These gaps leave both the reliability of the practical correction and the benefit of the proposed mechanism insufficiently established.

Questions for authors
  1. Does DOHF retain its GenEval advantage over NFT at the same inference-compute budget? Please provide a comparison under this constraint, state the NFT baseline’s guidance configuration, and clarify why comparable CFG is claimed to be unavailable to NFT.

  2. How accurately does the one-step estimator approximate the correction obtained from completed trajectories? A comparison on the Gaussian task across several noise levels would help establish whether the practical estimator supports the theoretical explanation.

  3. Does the (h)-dependent normalization improve DOHF when the rest of the method is unchanged? Please compare normalized and unnormalized corrections using the same estimator, training procedure, and sampling settings, with the overall correction strength tuned for both variants.

Reviewer_XJFTOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2(+1)
Presentation
2(+1)
Soundness
2
Read review
Paper summary

This paper introduces Diffusion Online h-guidance Fine-tuning (DOHF), a method for fine-tuning diffusion/flow-based generative models to sample from the desired terminal distribution. DOHF consists of two steps: (1) using Doob's h-transform to guide on-policy trajectories toward the target distribution, and (2) distilling the guided trajectories into the generative model by matching the velocity field. This online procedure progressively shifts the sampling distribution beyond the fixed pre-trained model, which can lead to rare but high-reward regions. Experiments under rare-reward, soft-constraint, and text-to-image alignment settings suggest that DOHF outperforms prior methods.

Critical strengths
  1. Aligning generative models to sample from the desired terminal distribution is challenging, especially when desirable outcomes are rare. DOHF addresses this challenge with on-policy Doob's h-guidance and distillation, an intuitive and well-motivated approach. The toy experiments: (1) Gaussian mixture with rewards targeting a minority mode and (2) contingency-table generation with soft constraints clearly and intuitively show the necessity of on-policy guidance and the distillation mechanism.

  2. The interpretation of DiffusionNFT [1] under Dooh's h-transform framework is interesting to me.

[1] Zheng, Kaiwen, et al. "Diffusionnft: Online diffusion reinforcement with forward process." International Conference on Learning Representations. Vol. 2026. 2026.

Critical weaknesses
  1. Limited methodological novelty. The main mechanism of DOHF consists of two parts: Doob's h-guidance and distillation. The estimator of the h-function is adopted from prior work [1] without an apparent modification. Distillation through regression with a guided target is an established technique [2, 3, 4]. Although the paper proposes a new combination in an on-policy fine-tuning framework, the method appears to incrementally extend existing approaches.

  2. Presentation of the paper could be improved. Introduction section would benefit from establishing the importance of generative model alignment and the challenges it poses, particularly when desirable outcomes are rare. Doob's h-transform and DOHF should be introduced as a method for addressing these challenges. This would make the motivation and problem formulation clearer. Also, separating the related work section would improve the organization and readability of the paper.

  3. Limited experimental results. The two toy experiments demonstrate the necessity of DOHF when the desired outcome is rare, but its effectiveness on real-world applications remains insufficiently validated. Tasks such as biological sequence generation with high reward and novelty would provide a more compelling test of this motivation.

[1] Zhu, Qijie, et al. "Training-Free Adaptation of Diffusion Models via Doob's $ h $-Transform." arXiv preprint arXiv:2602.16198 (2026).

[2] Meng, Chenlin, et al. "On distillation of guided diffusion models." 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2023.

[3] Domingo i Enrich, Carles, et al. "Adjoint matching: Fine-tuning flow and diffusion generative models with memoryless stochastic optimal control." International Conference on Learning Representations. Vol. 2025. 2025.

[4] Zhou, Wei, et al. "DanceOPD: On-Policy Generative Field Distillation." arXiv preprint arXiv:2606.27377 (2026).

Questions for authors
  1. DOHF defines the target at each iteration as w(x0)pold(x0)w(x_0)p^{\mathrm{old}}(x_0), with poldp^{\mathrm{old}} updated during training. This appears to differ from prior works maintaining a fixed target w(x0)ppre(x0)w(x_0)p^{\mathrm{pre}}(x_0), which retains the pretrained distribution as a reference. While updating the sampling distribution can help discover rare high-reward outcomes, why should the target distribution also change? Could the current model instead serve as a proposal distribution, with importance corrections used to preserve the fixed target w(x0)ppre(x0)w(x_0)p^{\mathrm{pre}}(x_0) [1]? How does DOHF preserve the quality, diversity, or other desirable properties of the pretrained model over successive updates?

  2. Estimating the h-function requires M rollout samples. How does DOHF compare with FlowGRPO and DiffusionNFT in total GPU-hours? Could the authors provide a computational cost analysis?

[1] Lee, Jaewoo, et al. "Diffusion alignment as variational expectation-maximization." International Conference on Learning Representations. Vol. 2026. 2026.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.