- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes BIDS, a score-based regularizer to mitigate reward hacking in inference-time noise optimization for diffusion models. BIDS uses a score-energy surrogate to limit divergence between optimized and base denoising trajectories, requires no model fine-tuning, and is extended to non-differentiable rewards. Experiments across multiple diffusion models and reward functions show improved prompt alignment, reduced distributional drift, and higher preference win rates compared to existing baselines.
Critical strengths
- The paper addresses reward hacking in inference-time noise optimization, where higher rewards can lead to unrealistic images or loss of prompt alignment.
- BIDS introduces a score-based regularizer without updating the diffusion model or training an additional network. It can be integrated into existing noise optimization methods.
- The experiments cover multiple diffusion backbones and reward functions. BIDS achieves better preference win rates and lower distributional drift than several baselines. The additional ablations and early-stopping comparisons are also useful.
Critical weaknesses
-
DNO already studies reward hacking and designed PRNO to keep optimized noise close to Gaussian statistics. DNO also supports non-differentiable rewards through gradient estimation. Therefore, the main contribution of BIDS seems to be limited.
-
The paper argues that noise-space regularization is insufficient because small noise perturbations can produce large image changes. However, this only demonstrates the limitation of noise-space Euclidean distance, not noise-distribution regularization. For a fixed diffusion sampler, the image distribution is the pushforward of the noise distribution. The paper does not clearly explain why trajectory-space regularization is necessary instead of improving noise-distribution regularization.
-
The paper defines KL divergence between optimized and base trajectory distributions, but the practical algorithm optimizes individual noise samples. In experiments, DDIM is not deterministic, but there are also single-step models. For deterministic sampling, it is unclear how this single-trajectory score-energy penalty represents the divergence between the full distributions induced by noise optimization.
Questions for authors
-
Given the paper's counterexample in Section 3 only shows the limitation of noise-space L2 distance, rather than noise-distribution regularization. Could the authors clarify why trajectory-space regularization is necessary instead of directly regularizing the noise distribution? Is it because trajectory regularization may allow for more effective noise variations, yielding higher rewards for a given level of divergence in the image distribution?
-
In deterministic DDIM, the same initial noise and prompt always produce the same trajectory and image. How are the optimized and base trajectory distributions defined? How does a penalty computed from a single pair of trajectories approximate the KL divergence between these distributions?
-
Why should the difference between squared score norms approximate the trajectory KL? Can two very different trajectories have the same score energy?
7 comments
Response (2/2)
Question 3: "Why should the difference between squared score norms approximate the trajectory KL? Can two very different trajectories have the same score energy?"
Response: We motivate the score-energy surrogate for trajectory KL using the relationship between expected squared score norms and KL divergence for Gaussian distributions with fixed variance.
Consider and . Here, is the base distribution, and changes its mean by without changing its variance. The base score is , so
The Gaussian KL formula gives
=\frac12\left(\mathbb E_q[s_p(x)^2]-\mathbb E_p[s_p(x)^2]\right).$$ Thus, in this fixed-variance Gaussian case, the difference in expected score energies is proportional to marginal KL. This motivates Eq. 5, which compares score energies before and after optimization across denoising steps to construct a computable surrogate for trajectory KL. **Different trajectories can indeed have the same score energy.** Squared score norms discard directional information. For example, under a two-dimensional standard Gaussian, the score is $s(x)=-x$. The points $(1,0)$ and $(0,1)$ both have squared score norm $1$. Although these points differ, they have the same base-model density. Furthermore, summing score energies across timesteps allows different per-step contributions to produce the same total. Therefore, equal score energy does not imply identical trajectories or guarantee small trajectory KL. Eq. 5 uses score energy as a practical surrogate rather than an exact characterization of trajectory divergence.Thank you
Thank you for the detailed clarification and additional empirical comparisons. I have no concerns about contribution or novelty now. However, my remaining question is the justification for using score-energy differences as a surrogate. As the authors acknowledge, different trajectories can have identical score energies. It is also unclear to me why score-energy differences should be preferred over other regularizers, like ? Is the choice more empirically motivated than theoretically justified as an approximation to trajectory KL? Thank you again!
Response (1/2)
We thank the reviewer for recognizing the practical contribution and the early-stopping comparisons.
Response: We agree with the reviewer that DNO addresses reward hacking through PRNO’s Gaussian noise regularization. As detailed in Appendix A.2, our DNO implementation already includes PRNO regularization. Despite this, Figure 3 shows that structural collapse persists under DNO, whereas BIDS preserves image structure.
BIDS instead regularizes denoising trajectories to limit image-space drift. Our key contribution is making this regularization tractable, using changes in the frozen model's score energy as a surrogate for trajectory KL (Eq. 5).
For non-differentiable rewards, DNO Hybrid-2 estimates reward gradients from perturbed samples, which may lead to reward hacking (as shown in Figure 6). BIDS uses the reward only to rank candidates, then differentiates a score-energy preference loss (Algorithm 2). This different update achieves a 63.56% preference win rate over Hybrid-2 with black-box Aesthetic Score across five seeds (Section 5.3).
These results demonstrate that BIDS makes image-distribution-aware regularization practical and effective in mitigating reward hacking, including for non-differentiable rewards.
Response: Yes. The motivation behind trajectory-space regularization is to allow more effective noise variations that improve reward while preserving the generated content. PRNO’s regularizer penalizes deviations from Gaussian noise statistics but does not capture how noise updates propagate through the sampler and affect the generated images. BIDS addresses this limitation by using score-energy changes along the denoising trajectory as a tractable surrogate for trajectory KL, thereby accounting for the sampler's response to noise perturbations.
Figure 13 provides empirical support for this motivation: BIDS achieves higher Darkness Reward at comparable CLIPScore, indicating better reward optimization while maintaining prompt alignment, even when DNO's regularization strength and stopping step are tuned. Appendix A.4 further shows analytically that, in the linear case, the noise-space penalty corresponding to image-space KL depends on the sampler.
Response: A deterministic sampler still induces a distribution over trajectories because the initial noise is sampled from a Gaussian distribution. For deterministic DDIM, let Gθ(z,c) denote the trajectory generated from initial noise z. The base trajectory distribution is induced by z∼N(0,I). The optimized trajectory distribution is similarly induced by applying the noise-optimization procedure to each initial noise sample before generating its trajectory.
Like DNO, BIDS optimizes individual noise samples at inference time. The optimized distribution is defined implicitly by the mapping from each initial noise sample to its optimized noise, obtained by applying the same optimization procedure. Thus, the population-level formulation describes the distribution induced by repeating the algorithm over different initial noise draws, whereas the practical algorithm operates on individual samples.
Equation (5) defines a population-level score-energy surrogate for trajectory KL. Algorithm 1 regularizes individual noise updates with the paired penalty ∣S0−S∣, computed from the base and optimized trajectories generated from the same initial noise.
By Jensen's inequality,
∣Ezinit[S0−S]∣≤Ezinit[∣S0−S∣].
Thus, averaging Algorithm 1's paired penalties over initial noise draws upper-bounds the population score-energy penalty in Eq. 5.
For single-step models, the trajectory contains only one denoising step. BIDS therefore compares the squared norms of the model's predictions before and after noise optimization (Appendix A.2).