- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes DOHF, an online diffusion fine-tuning framework based on Doob's h-transform. It expresses a terminal reward-weighted distribution through a local score or velocity correction, estimates this correction using sampled transitions and reward evaluations, and distills it into the model through regression. The paper also characterizes the population-level relationship to DiffusionNFT, identifying a state-dependent scaling that recovers the ideal correction. Experiments cover Gaussian rare-mode adaptation, constrained contingency-table generation, and SD3.5-Medium text-to-image fine-tuning, with improvements over the base model and several adaptation baselines.
Critical strengths
-
Useful theoretical interpretation. The connection between terminal distribution tilting, local velocity correction, and DiffusionNFT's population optimum is informative. Under the idealized assumptions, the derivation appears sound and explains the role of state-dependent guidance strength. Its main value is a clearer interpretation of existing update mechanisms, which I consider a meaningful contribution.
-
Practical realization with supporting evidence. Directly regressing the estimated correction provides a concrete fine-tuning procedure compatible with non-differentiable rewards, without training a separate critic. The statistical and text-to-image experiments demonstrate practical utility beyond the theoretical interpretation and support a positive assessment.
Critical weaknesses
-
The connection between the ideal theory and the practical reward estimator needs stronger justification. The theoretical correction depends on h(x_t,t,c) = E[w(X_0) | X_t=x_t,c], whereas Section 3.1 replaces complete continuations with a one-step terminal prediction and evaluates the terminal reward on that prediction. A predicted clean image need not receive a score representative of its eventual completion, especially at high noise levels. This affects the reward weights and consequently the estimated correction, rather than merely adding sampling variance; increasing the number of candidates alone does not remove the surrogate bias. Relevant prior work, SPO, adapts PickScore to timestep-dependent denoised predictions (Section 5.1, https://arxiv.org/html/2406.04314v1), reinforcing the need to validate this assumption. This does not invalidate the h-transform identity or establish that DOHF fails, but it limits what that identity guarantees for the implemented algorithm. The paper should distinguish its exact population-level characterization from its approximate estimator and finite-step training procedure, with evidence that the surrogate preserves useful reward ordering or correction directions across timesteps.
-
A controlled ablation of the state-dependent normalization is missing. Section 3.3 emphasizes the distinction between the normalized correction, grad log h, and the h-scaled correction associated with DiffusionNFT, arguing that normalization avoids attenuating updates in low-h regions. However, the method-level comparisons do not isolate this mechanism. A useful ablation would compare the normalized and unnormalized estimators within the same regression framework, using the same rollout protocol, reward estimator, and training budget, while tuning a global correction scale for each variant. This would distinguish the value of state-dependent scaling from a change in overall update magnitude. An analysis of progress and estimator variability in low-h regions would further test the specific mechanism proposed by the theory. This matters because recovering the ideal correction does not by itself establish an empirical advantage under finite-sample estimation.
Questions for authors
-
Which exactness claims remain valid after introducing the one-step surrogate, finite-sample normalization, and approximate regression? What additional assumptions are required?
-
Can a small diagnostic compare one-step predictions with complete continuations across noise levels, measuring reward-ranking agreement or agreement between the resulting correction directions?
-
Can the authors isolate state-dependent normalization through a controlled ablation against an unnormalized correction with a tuned global scale, and report whether its benefit is concentrated in low-h regions as suggested by Section 3.3?