acceptodds
Under review as a conference paper at ICLR 2027

Steepest Guidance: A Practical and Principled Approach to Inference-Time Alignment of Flow and Diffusion-based Models

Abstract

Inference-time alignment of flow and diffusion-based models is critical for achieving flexible generative modeling. Theoretically, Doob's -transform provides an elegant solution to this problem, and most existing methods are based on this principle. However, in practice, estimating the optimal guidance derived from Doob's -transform at inference time is challenging. To deal with this issue, we regard inference-time alignment as a sequential optimization problem in the space of probability measures and propose a novel framework called *Steepest Guidance*, based on the principle of maximizing local improvement in the objective. We provide a theoretical analysis of the proposed method and demonstrate its effectiveness through extensive experiments.

Primary area:
generative models
Keywords:
Diffusion model, Alignment, Optimization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_wJb2OpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper formulates inference-time alignment as steepest ascent of a reward functional over the time-indexed distributions of a pretrained flow or diffusion model. For expected reward, the resulting velocity correction can be written using a conditional expectation and estimated by sampling future states. The paper also derives a particle-based first variation for nonlinear population objectives, including risk-sensitive and diversity-aware examples, and proposes a regularized variant. Experiments with Stable Diffusion and FLUX evaluate three prompts and four differentiable rewards, followed by smaller demonstrations of CVaR optimization and diversity control.

The conceptual contribution is to place several guidance procedures in a common functional-gradient view and to distinguish local improvement of intermediate distributions from the terminal-value Doob-transform perspective. The practical algorithm, however, replaces the exact conditional quantities in the theory with finite lookahead samples and an approximate posterior map.

Critical strengths
  1. The time-dependent distributional formulation is clear and, in my view, the strongest part of the paper. The derivation explains what direction is being optimized at each time and makes the relation between linear expected-reward guidance and nonlinear population objectives easier to see.

  2. The method produces consistent reward improvements across two fairly different pretrained generators and several reward models. The toy experiments and some of the ablations are useful for understanding why future-state sampling helps over a purely local gradient.

Critical weaknesses
  1. The guarantees apply to an ideal continuous-time population update with exact conditional quantities, while the implemented method uses Diamond Map/Tweedie approximations, a small number of lookahead samples, Euler integration, and finite particle batches. For the nonlinear case, the empirical update is even further from the object analyzed in the convergence theorem. No experiment measures the bias of these approximations or tests whether the claimed ascent property holds along actual trajectories. As a result, I like the formulation but am not yet convinced that the practical method inherits its advertised guarantees.

  2. The empirical comparison is difficult to interpret without a matched compute budget. The methods use different amounts of lookahead sampling, reward evaluation, and backpropagation; the reported SALD setting also removes a costly component of the original procedure. There is no wall-clock or NFE-normalized comparison to the closest greedy/gradient-guidance variants. Evidence for the nonlinear objectives is especially preliminary: the CVaR result has no strong matched baseline, and the diversity example is based on a small batch while optimizing and evaluating closely related features.

Questions for authors
  1. How large is the error between the exact conditional direction in the derivation and the direction produced by the actual posterior approximation and finite lookahead sampler? Can this be measured during sampling, rather than only on the toy problem?

  2. Can the authors provide a compute-matched comparison, including wall-clock time, number of reward-model calls, and backward passes, against full SALD and the closest greedy or gradient-guidance baseline?

  3. For the nonlinear objectives, can the authors add a matched baseline and evaluate sample quality independently of the feature or reward used for optimization?

4 comments
AuthorsOpenReview ↗

Response to Reviewer wJb2

Thank you for your helpful feedback and constructive suggestions. We address the main concerns below. Please refer to the general response above for a gap between theory and practice, and computational cost.

W1: Gap between theory and practice

We acknowledge that there is a gap between the theory and practice of our method. However, theoretical analysis provides valuable insights into the behavior of our method and guides the design of practical algorithms. We have also conducted extensive empirical evaluations to validate the effectiveness of our approach in real-world scenarios.

W2&Q3: Baseline for nonlinear objectives

The baselines considered in our paper are designed for expected-reward maximization with a fixed per-sample reward. They are not directly applicable to general nonlinear reward functionals, whose first variations depend on the generated distribution. Extending these methods to such objectives requires additional algorithmic choices, including how to estimate this distribution-dependent reward signal during generation. For diversity-seeking generation, entropy regularization is independent of the feature used in evaluation.

Q1: Bias in text-image experiments

We would like to clarify that we do not claim that the proposed method is completely unbiased in practice. Our main contribution is to provide a principled framework for guidance that is unbiased in the idealized setting and to demonstrate its practical effectiveness through empirical evaluations. Since computing the exact direction of the Doob's h-transform is intractable for its non-linearlity and heavy computational cost of exact sampling, we have reported the bias in toy experiments where the exact posterior is known.

Q2: comparison with full SALD and gradient-based baselines

Although some of the reward functions used in our experiments are differentiable, we adopt them as standard benchmarks for reward optimization. Our method targets the black-box setting, where only reward values are accessible and reward gradients are unavailable. We therefore focus our comparisons on empirical Doob guidance, SVDD, and SALD, which operate under the same derivative-free access setting.

For SALD, we have conducted additional experiments for full SALD with slowdown as in the original paper and provided the results in the appendix (the right panel of Fig. 9). We do not see a significant advantage of full SALD over SALD without slowdown in our previous experiments.

If you have any remaining concerns, we would be happy to address them. We would appreciate it if you could reconsider your score in light of these clarifications and the additional experimental results.

Dear Reviewer wJb2,

Thank you again for your time and thoughtful feedback. As the discussion period draws to a close, please let us know if you have any remaining questions or concerns. We would be happy to address them and provide any further clarification.

Best regards, The Authors

I thank the authors for their response. While I recognize the paper's theoretical contribution, the experimental evidence remains limited and relies too heavily on toy settings to support the claims convincingly. I will therefore maintain my current score.

Thank you for your follow-up and for recognizing the theoretical contribution. We would like to clarify the role of the toy experiments in the paper.

Our main contribution is the distributional optimization formulation of inference-time alignment and the resulting derivation of guidance. The unbiasedness of the estimator for linear reward functionals under exact posterior sampling is one consequence of this framework. The toy experiments are designed to verify this property empirically.

The practical effectiveness of the proposed algorithm is evaluated separately through text-to-image experiments. The revised Table 1 extends this evaluation to a more diverse set of prompts from DrawBench.

Reviewer_WkcvOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies training-free, inference-time reward alignment of flow and diffusion models. It treats alignment as a sequential optimization problem over probability measures. Prop. 3.1 gives an exact decomposition of the terminal reward improvement and the path KL under an arbitrary guidance field. From this, the authors derive steepest guidance, gt=λσt2∇yδVδμ(t,πtg)g _t = \lambda\sigma _t^2 \nabla _y \frac{\delta V}{\delta\mu}(t,\pi_t^g), which maximizes the local improvement at each time. This replaces the Doob hh-transform guidance σt2∇log⁡E[exp⁡(λr(Y1))∣Yt=y]\sigma _t^2\nabla\log\mathbb{E}[\exp(\lambda r(Y_1))\mid Y_t=y]. For linear rewards, the guidance reduces to λσt2∇yE[r(Y1)∣Yt=y]\lambda\sigma _t^2\nabla _y\mathbb{E}[r(Y_1)\mid Y _t=y], which admits an unbiased REINFORCE estimator from exact posterior lookahead samples. Doob-based Monte Carlo estimators, by contrast, are biased (Fig. 1). Non-linear functionals (CVaR, Rao's quadratic entropy, entropy) are handled with particle plug-in estimates of the first variation. A regularized variant targets a terminal-KL objective by inflating the diffusion noise (Prop. 4.2).

On the theory side, the paper proves exact reward improvement for the ideal dynamics (Thm. 3.2), improvement of the KL-regularized objective (Cor. 4.1, Thm. 4.3), and global convergence for concave functionals under a uniform log-Sobolev inequality and boundedness assumptions (Thm. 4.5). Experiments include a 1D Gaussian-mixture toy problem, and SD v1.5 and FLUX.1-dev on three prompts with Blueness, ImageReward, PickScore and Compressibility rewards. Baselines are DOIT (REINFORCE Doob) and SVDD. The appendix adds SALD, a guidance-scaling ablation for Doob, an ablation over kk, CVaR results, and a single-prompt diversity experiment with Rao's entropy.

Critical strengths
  1. A clean and general framework for non-linear reward functionals. Prop. 3.1 and the resulting steepest-guidance rule handle CVaR, Rao's entropy and entropy regularization in one training-free, derivative-free scheme. Most Doob-based guidance cannot do this, because it requires the first variation at the unknown optimal distribution. The convergence analysis (Thm. 4.5) extends mean-field Langevin arguments to an objective that changes along the generative process, which is a non-trivial theoretical contribution. This generality is the main reason the paper is of interest.

  2. A correct diagnosis of the bias in Doob-based Monte Carlo guidance. The paper shows that REINFORCE estimators of Doob guidance can be strongly biased even where the plug-in estimator is unbiased (Fig. 1, App. L.2). It proposes an estimator that is unbiased for linear rewards under exact posterior sampling. This is a meaningful conceptual point for the growing line of training-free guidance methods.

Critical weaknesses

1. The experiments do not test the paper's central claim, because they never operate in the regime where the unbiasedness argument applies.

The paper's main argument for steepest over Doob guidance is unbiasedness given exact samples from π1(⋅∣Yt=y)\pi_1(\cdot\mid Y_t=y). The experiments use no such sampler:

  • For SD, the lookahead samples come from a one-step Tweedie estimate with an inflated noise ratio of 5.0 (App. L.4). This is not a posterior sampler.
  • For FLUX, they come from an approximate Diamond Map.
  • The bias of the steepest estimator under these samplers is never measured. Fig. 1 uses exact conditional samples in a 1D Gaussian.

2. The empirical evaluation is too narrow to support the claim of consistent improvement.

  • All text-to-image results use three hand-picked prompts with 32 images each (Tables 1, 5, 6). λ is grid-searched separately for each prompt, reward, model and method (Table 4). The diversity experiment uses one prompt and 8 images (Table 7).
  • The gains are clear on the low-level rewards (Blueness, Compressibility), but small or absent on the human-preference rewards. On SD, SVDD obtains a higher PickScore than steepest guidance on two of three prompts (22.95 vs. 22.90; 22.57 vs. 22.54). Most ImageReward and PickScore differences are well within one standard deviation. "Consistently outperforms" is therefore overstated.
  • There is no compute comparison. Each guided step requires kk lookahead samples and kk reward evaluations, while SVDD's particle budget and the wall-clock or NFE costs are not reported.
  • Best-of-N and gradient-based guidance (DPS / universal guidance), both standard and cheap baselines for differentiable rewards, are missing.
  • There is no held-out quality metric to detect reward hacking. The regularized variant's KL control is assessed only through ImageReward as a proxy (Fig. 3).
  • The theory (Thm. 3.2, 4.5) assumes the exact steepest guidance, a uniform LSI, and bounded Φt,Ψt\Phi_t, \Psi_t involving the unknown ratio πt∗/πt\pi_t^*/\pi_t. These assumptions cannot be checked for image models, so the guarantees say little about the implemented algorithm.
Questions for authors
  1. Bias under the samplers actually used. Can you repeat the bias analysis of Fig. 1 with the samplers used in the experiments (Tweedie with noise ratio 5.0 for SD, Diamond Map for FLUX), or in a setting where the exact posterior is known? How large is the bias of the steepest estimator there, compared with the Doob estimator?
  2. Linear vs. softmax weighting. Since steepest and Doob guidance share the same REINFORCE score and differ only in linear versus softmax weights, how does the performance gap behave as λ decreases, where the two coincide to first order? Is the gap explained by the weight variance or saturation of the softmax at large λ?
  3. Scale and compute. Can you report results on a standard prompt set (≥100 prompts, e.g., HPDv2 or DrawBench), with λ fixed per reward rather than per prompt? Please compare against SVDD and Best-of-N under matched reward evaluations or matched wall-clock time, and report SVDD's particle budget and the runtime of each method.
  4. Gradient-based baselines. For the differentiable rewards (ImageReward, PickScore, Blueness), how does steepest guidance compare with DPS / universal guidance at a similar cost?
  5. Reward hacking. Can you report held-out metrics (e.g., a preference model not used for guidance, CLIPScore, or FID/KID against reference images) to show that the reward gains do not come at the cost of image quality or prompt fidelity?
4 comments
AuthorsOpenReview ↗

Response to Reviewer Wkcv

Thank you for your helpful feedback and constructive suggestions. We address the main concerns below. Please also refer to the general response above.

W1&Q1: Bias in text-to-image experiments

We would like to clarify that we do not claim that the proposed method is completely unbiased in practice. Our main contribution is to provide a principled framework for guidance that is unbiased in the idealized setting and to demonstrate its practical effectiveness through empirical evaluations. Since computing the exact direction of the Doob's h-transform is intractable for its non-linearlity and heavy computational cost of exact sampling, we have reported the bias in toy experiments where the exact posterior is known.

W2&Q3: Experimental Evaluation

We have conducted additional large-scale experiments with 100 DrawBench prompts using the same λ\lambda across prompts. The following table reports the mean and standard error for each metric.

ModelRewardUnguidedSteepest (Ours)DOITSVDD
SDBlueness-0.50 ± 0.0050.05 ± 0.010-0.47 ± 0.005-0.43 ± 0.004
SDImageReward0.23 ± 0.040.42 ± 0.040.26 ± 0.040.31 ± 0.04
SDPickScore21.18 ± 0.0321.33 ± 0.0321.21 ± 0.0321.28 ± 0.03
SDCompressibility-11.73 ± 0.10-4.07 ± 0.06-11.35 ± 0.10-10.44 ± 0.09
FLUXBlueness-0.56 ± 0.007-0.05 ± 0.006-0.46 ± 0.007-0.36 ± 0.006
FLUXImageReward0.89 ± 0.031.51 ± 0.031.15 ± 0.021.36 ± 0.02
FLUXPickScore22.29 ± 0.0323.14 ± 0.0322.68 ± 0.0323.05 ± 0.03
FLUXCompressibility-6.35 ± 0.04-4.36 ± 0.03-5.89 ± 0.03-5.28 ± 0.03

The results are consistent with previous findings, demonstrating the effectiveness of our method even with a shared hyperparameter.

For computational cost, the number of lookahead samples and reward evaluations is kept consistent across methods to ensure a fair comparison. We have emphasized this point in Section 5.2 of the revised version of the paper. In addition, we have added wall-clock time comparisons in the appendix L.6.

W2: Gap between theory and practice

We acknowledge that there is a gap between the theory and practice of our method. However, theoretical analysis provides valuable insights into the behavior of our method and guides the design of practical algorithms. We have also conducted extensive empirical evaluations to validate the effectiveness of our approach in real-world scenarios.

W2&Q4: Gradient-based baselines

Although some of the reward functions used in our experiments are differentiable, we adopt them as standard benchmarks for reward optimization. Our method targets the black-box setting, where only reward values are accessible and reward gradients are unavailable. We therefore focus our comparisons on empirical Doob guidance, SVDD, and SALD, which operate under the same derivative-free access setting.

W2&Q5: Reward hacking

We acknowledge that optimizing a reward can involve a trade-off with image quality and prompt fidelity. However, Table 1 addresses the more basic question of whether each method can effectively optimize the target reward. Existing methods struggle to improve the reward sufficiently to realize even this trade-off in our experiments. Path KL provides an upper bound on terminal KL, but the bound can be loose. Instead, we report CLIPScore for the same images in Table 6 in the appendix. The scores remain within a reasonable range relative to unguided sampling.

In addition, we proposed regularized steepest guidance to mitigate reward hacking and report the trade-off between the optimized reward and image quality in Fig. 3. We have added the results of empirical Doob's transform and SVDD to this figure for comparison. Our method achieves a better reward–quality trade-off, attaining substantially higher Blueness while keeping ImageReward close to the unguided baseline.

Q2: Linear vs. softmax weighting.

For sufficiently large kk, as λ\lambda decreases, two weighting schemes become similar and the performance of both methods approaches that of unguided sampling. Therefore, the regime we are interested in is the one where λ\lambda is not too small, and in this regime, the difference between linear and softmax weighting becomes significant. In addition, if k=1k = 1, softmax weighting yields E[gsoftmax]=0E[g_{\mathrm{softmax}}] = 0 while linear weighting still provides a non-zero expected guidance.

If you have any remaining concerns, we would be happy to address them. We would appreciate it if you could reconsider your score in light of these clarifications and the additional experimental results.

Dear Reviewer Wkcv,

Thank you again for your time and thoughtful feedback. As the discussion period draws to a close, please let us know if you have any remaining questions or concerns. We would be happy to address them and provide any further clarification.

Best regards, The Authors

I thank the authors for the detailed response and the additional experiments. The new DrawBench results address my main concern about the narrow evaluation. The black-box framing is a reasonable justification for restricting comparisons to derivative-free methods. Adding Doob and SVDD to the trade-off plot in Fig. 3 is helpful, and I appreciate it. Held-out quality evidence is still limited to CLIPScore on the original prompts. It is not reported for the new large-scale DrawBench setting.

Thank you for your follow-up and for acknowledging the additional experiments.

To clarify, Table 6 in Appendix L.5 reports CLIPScore for the DrawBench evaluation, not for the original prompts.

If this clarification addresses your remaining concern, we would appreciate it if you could reconsider your score.

Reviewer_LgPrOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Steepest Guidance, a training-free method for inference-time reward alignment of flow and diffusion models. The authors point out the problems with existing methods based on Doob's h-transform. Instead of estimating the globally optimal guidance, the proposed method moves, at each time step, in the direction that improves the reward fastest (a "steepest ascent" step), with a path-KL penalty. The paper proves a reward-improvement identity and a global convergence result for concave reward functionals, and evaluates the method on SD1.5 and FLUX with four rewards.

Critical strengths
  1. The idea is simple, well motivated, and easy to follow. Replacing ∇log⁡E[exp⁡(λr)]\nabla\log\mathbb{E}[\exp(\lambda r)] with λ∇E[r]\lambda\nabla\mathbb{E}[r] makes the estimator linear in the samples, so it is unbiased when exact posterior samples are available.
  2. The method needs no reward gradient and no backpropagation through the sampling process. It only uses the conditional score of the posterior, which is available from the pretrained model (Lemma 2.2). Therefore, it can be used with non-differentiable rewards.
Critical weaknesses
  1. The main results report only the reward that is optimized. It is hard to tell whether the higher reward comes from better alignment or from reward hacking.
  2. The evaluation is small. The image experiments use only few prompts with 32 images each, and λ\lambda is tuned separately for each prompt. It is unclear how the method performs on a standard prompt set with a single shared λ\lambda.
  3. There is a gap between the theory and the practice. The analysis is in continuous time and assumes the exact first variation and exact posterior samples. There is no error analysis for time discretization or for the Monte Carlo estimation.
Questions for authors
  1. Could you evaluate the method on a larger prompt set with one λ\lambda shared across prompts?
  2. Could you also report metrics that are not optimized, together with the path KL of each method?
2 comments
AuthorsOpenReview ↗

Response to Reviewer LgPr

Thank you for your helpful feedback and constructive suggestions. We address the main concerns below. Please also refer to the general response above.

W1&Q2: On reward hacking

As you pointed out, optimizing a reward can involve a trade-off with image quality and prompt fidelity. However, Table 1 addresses the more basic question of whether each method can effectively optimize the target reward. Existing methods struggle to improve the reward sufficiently to realize even this trade-off in our experiments. Path KL provides an upper bound on terminal KL, but the bound can be loose. Instead, we report CLIPScore for the same images in Table 6 in the appendix. The scores remain within a reasonable range relative to unguided sampling.

In addition, we proposed regularized steepest guidance to mitigate reward hacking and report the trade-off between the optimized reward and image quality in Fig. 3. We have added the results of empirical Doob's transform and SVDD to this figure for comparison. Our method achieves a better reward–quality trade-off, attaining substantially higher Blueness while keeping ImageReward close to the unguided baseline.

W2&Q1: Large-Scale Experiments

We have conducted additional large-scale experiments with 100 DrawBench prompts using the same λ\lambda across prompts. The following table reports the mean and standard error for each metric.

ModelRewardUnguidedSteepest (Ours)DOITSVDD
SDBlueness-0.50 ± 0.0050.05 ± 0.010-0.47 ± 0.005-0.43 ± 0.004
SDImageReward0.23 ± 0.040.42 ± 0.040.26 ± 0.040.31 ± 0.04
SDPickScore21.18 ± 0.0321.33 ± 0.0321.21 ± 0.0321.28 ± 0.03
SDCompressibility-11.73 ± 0.10-4.07 ± 0.06-11.35 ± 0.10-10.44 ± 0.09
FLUXBlueness-0.56 ± 0.007-0.05 ± 0.006-0.46 ± 0.007-0.36 ± 0.006
FLUXImageReward0.89 ± 0.031.51 ± 0.031.15 ± 0.021.36 ± 0.02
FLUXPickScore22.29 ± 0.0323.14 ± 0.0322.68 ± 0.0323.05 ± 0.03
FLUXCompressibility-6.35 ± 0.04-4.36 ± 0.03-5.89 ± 0.03-5.28 ± 0.03

The results are consistent with previous findings, demonstrating the effectiveness of our method even with a shared hyperparameter.

W3: Gap between theory and practice

We acknowledge that there is a gap between the algorithm used in practice and the theory. However, the practical implementation can be regarded as an approximation to the idealized continuous-time update and thus, analyzing its behavior can provide insights into the performance of the practical method. Most of the existing works (e.g. [Zhu et al. (2026)], [Dandapanthula & Boffi (2026)] ) also analyze idealized updates rather than their practical implementations.

In addition, we do not claim that our method is completely free from bias. Our claim is that the Doob's h-transform suffers from estimation bias even when the sampling is exact, while our proposed method avoids this type of bias. Roughly speaking, If sampling is not exact, the total bias can be decomposed as (Total Bias)=(Estimation Bias)+(Approximate Sampling Bias)(\text{Total Bias}) = (\text{Estimation Bias}) + (\text{Approximate Sampling Bias}) for linear functionals. While the existing methods based on Doob's h-transform suffer from both types of bias, our proposed method suffers from only approximate sampling bias. Indeed, our numerical experiments demonstrate that our proposed method exhibits better performance even when the sampling is not exact. We emphasize these points in Remark 3.3 in the revised manuscript.

If you have any remaining concerns, we would be happy to address them. We would appreciate it if you could reconsider your score in light of these clarifications and the additional experimental results.

Dear Reviewer LgPr,

Thank you again for your time and thoughtful feedback. As the discussion period draws to a close, please let us know if you have any remaining questions or concerns. We would be happy to address them and provide any further clarification.

Best regards, The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

Thanks for the valuable feedback from the reviewers. We appreciate the positive comments on the motivation and formulation of our work. As noted by the reviewers, our main contribution lies in the clean formulation of the inference-time alignment based on the distributional optimization and the simple and principled algorithm design.

Concerns raised by the reviewers include the connection between theory and practice and the experimental evaluation of our method. We have conducted additional experiments and revised the manuscript to address these concerns. The revised PDF is uploaded to OpenReview and Substantive changes are marked in red. Here, we summarize the main concerns raised by the reviewers and our responses to them.

Connection between Theory and Practice

As reviewers pointed out, there is a gap between the algorithm used in practice and the theory. However, the practical implementation can be regarded as an approximation to the idealized continuous-time update and thus, analyzing its behavior can provide insights into the performance of the practical method. Most of the existing works (e.g. [Zhu et al. (2026)], [Dandapanthula & Boffi (2026)] ) also analyze idealized updates rather than their practical implementations.

In addition, we do not claim that our method is completely free from bias. Our claim is that the Doob's h-transform suffers from estimation bias even when the sampling is exact, while our proposed method avoids this type of bias. Roughly speaking, If sampling is not exact, the total bias can be decomposed as (Total Bias)=(Estimation Bias)+(Approximate Sampling Bias)(\text{Total Bias}) = (\text{Estimation Bias}) + (\text{Approximate Sampling Bias}) for linear functionals. While the existing methods based on Doob's h-transform suffer from both types of bias, our proposed method suffers from only approximate sampling bias. Indeed, our numerical experiments demonstrate that our proposed method exhibits better performance even when the sampling is not exact. We emphasize these points in Remark 3.3 in the revised manuscript.

Evaluation on Larger Prompt Sets

We replaced Table 1 with results on 100 DrawBench prompts using the same λ\lambda across prompts. The results are consistent with previous findings, demonstrating the effectiveness of our method even with a shared hyperparameter.

Compute-time Comparison

In our experiments, the number of lookahead samples and reward evaluations is kept consistent across methods to ensure a fair comparison. We have emphasized this point in Section 5.2 of the revised version of the paper. In addition, we have added wall-clock time comparisons in the appendix L.6.

Reward Hacking

Optimizing a reward can involve a trade-off with image quality and prompt fidelity. However, Table 1 addresses the more basic question of whether each method can effectively optimize the target reward. Existing methods struggle to improve the reward sufficiently to realize even this trade-off in our experiments. We also report CLIPScore for the same images in Table 6 in the appendix. The scores remain within a reasonable range relative to unguided sampling.

In addition, we proposed regularized steepest guidance to mitigate reward hacking and report the trade-off between the optimized reward and image quality in Fig. 3. We have added the results of empirical Doob's transform and SVDD to this figure for comparison. Our method achieves a better reward–quality trade-off, attaining substantially higher Blueness while keeping ImageReward close to the unguided baseline.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.