- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper formulates inference-time alignment as steepest ascent of a reward functional over the time-indexed distributions of a pretrained flow or diffusion model. For expected reward, the resulting velocity correction can be written using a conditional expectation and estimated by sampling future states. The paper also derives a particle-based first variation for nonlinear population objectives, including risk-sensitive and diversity-aware examples, and proposes a regularized variant. Experiments with Stable Diffusion and FLUX evaluate three prompts and four differentiable rewards, followed by smaller demonstrations of CVaR optimization and diversity control.
The conceptual contribution is to place several guidance procedures in a common functional-gradient view and to distinguish local improvement of intermediate distributions from the terminal-value Doob-transform perspective. The practical algorithm, however, replaces the exact conditional quantities in the theory with finite lookahead samples and an approximate posterior map.
Critical strengths
-
The time-dependent distributional formulation is clear and, in my view, the strongest part of the paper. The derivation explains what direction is being optimized at each time and makes the relation between linear expected-reward guidance and nonlinear population objectives easier to see.
-
The method produces consistent reward improvements across two fairly different pretrained generators and several reward models. The toy experiments and some of the ablations are useful for understanding why future-state sampling helps over a purely local gradient.
Critical weaknesses
-
The guarantees apply to an ideal continuous-time population update with exact conditional quantities, while the implemented method uses Diamond Map/Tweedie approximations, a small number of lookahead samples, Euler integration, and finite particle batches. For the nonlinear case, the empirical update is even further from the object analyzed in the convergence theorem. No experiment measures the bias of these approximations or tests whether the claimed ascent property holds along actual trajectories. As a result, I like the formulation but am not yet convinced that the practical method inherits its advertised guarantees.
-
The empirical comparison is difficult to interpret without a matched compute budget. The methods use different amounts of lookahead sampling, reward evaluation, and backpropagation; the reported SALD setting also removes a costly component of the original procedure. There is no wall-clock or NFE-normalized comparison to the closest greedy/gradient-guidance variants. Evidence for the nonlinear objectives is especially preliminary: the CVaR result has no strong matched baseline, and the diversity example is based on a small batch while optimizing and evaluating closely related features.
Questions for authors
-
How large is the error between the exact conditional direction in the derivation and the direction produced by the actual posterior approximation and finite lookahead sampler? Can this be measured during sampling, rather than only on the toy problem?
-
Can the authors provide a compute-matched comparison, including wall-clock time, number of reward-model calls, and backward passes, against full SALD and the closest greedy or gradient-guidance baseline?
-
For the nonlinear objectives, can the authors add a matched baseline and evaluate sample quality independently of the feature or reward used for optimization?
4 comments
Dear Reviewer wJb2,
Thank you again for your time and thoughtful feedback. As the discussion period draws to a close, please let us know if you have any remaining questions or concerns. We would be happy to address them and provide any further clarification.
Best regards, The Authors
I thank the authors for their response. While I recognize the paper's theoretical contribution, the experimental evidence remains limited and relies too heavily on toy settings to support the claims convincingly. I will therefore maintain my current score.
Thank you for your follow-up and for recognizing the theoretical contribution. We would like to clarify the role of the toy experiments in the paper.
Our main contribution is the distributional optimization formulation of inference-time alignment and the resulting derivation of guidance. The unbiasedness of the estimator for linear reward functionals under exact posterior sampling is one consequence of this framework. The toy experiments are designed to verify this property empirically.
The practical effectiveness of the proposed algorithm is evaluated separately through text-to-image experiments. The revised Table 1 extends this evaluation to a more diverse set of prompts from DrawBench.
Response to Reviewer wJb2
Thank you for your helpful feedback and constructive suggestions. We address the main concerns below. Please refer to the general response above for a gap between theory and practice, and computational cost.
W1: Gap between theory and practice
We acknowledge that there is a gap between the theory and practice of our method. However, theoretical analysis provides valuable insights into the behavior of our method and guides the design of practical algorithms. We have also conducted extensive empirical evaluations to validate the effectiveness of our approach in real-world scenarios.
W2&Q3: Baseline for nonlinear objectives
The baselines considered in our paper are designed for expected-reward maximization with a fixed per-sample reward. They are not directly applicable to general nonlinear reward functionals, whose first variations depend on the generated distribution. Extending these methods to such objectives requires additional algorithmic choices, including how to estimate this distribution-dependent reward signal during generation. For diversity-seeking generation, entropy regularization is independent of the feature used in evaluation.
Q1: Bias in text-image experiments
We would like to clarify that we do not claim that the proposed method is completely unbiased in practice. Our main contribution is to provide a principled framework for guidance that is unbiased in the idealized setting and to demonstrate its practical effectiveness through empirical evaluations. Since computing the exact direction of the Doob's h-transform is intractable for its non-linearlity and heavy computational cost of exact sampling, we have reported the bias in toy experiments where the exact posterior is known.
Q2: comparison with full SALD and gradient-based baselines
Although some of the reward functions used in our experiments are differentiable, we adopt them as standard benchmarks for reward optimization. Our method targets the black-box setting, where only reward values are accessible and reward gradients are unavailable. We therefore focus our comparisons on empirical Doob guidance, SVDD, and SALD, which operate under the same derivative-free access setting.
For SALD, we have conducted additional experiments for full SALD with slowdown as in the original paper and provided the results in the appendix (the right panel of Fig. 9). We do not see a significant advantage of full SALD over SALD without slowdown in our previous experiments.
If you have any remaining concerns, we would be happy to address them. We would appreciate it if you could reconsider your score in light of these clarifications and the additional experimental results.