acceptodds
Under review as a conference paper at ICLR 2027

Smooth Then Balance: Gradient Smoothing and Preference Anchoring for Multi-Objective Diffusion RL

Abstract

Multi-objective diffusion reinforcement learning (RL) fine-tunes a single diffusion model to jointly improve multiple objectives (e.g., image quality, human preference, and text rendering). Gradient-based multi-objective optimization methods such as the Multiple Gradient Descent Algorithm (MGDA) avoid manually designed reward weights by combining the gradients of individual rewards into a common descent direction for all rewards. However, we empirically find two drawbacks of MGDA when it is applied to diffusion RL. First, the reward gradients are highly noisy, which makes the resulting update directions unreliable. Second, MGDA determines the weights solely from the gradient directions and can yield highly imbalanced weights that leave some rewards under-optimized. To address these problems, we propose Smooth Then Balance (STB), which first smooths each reward gradient with an exponential moving average before the weights are computed, and then balances the smoothed gradients with a preference anchor that guarantees every reward a minimum weight. Our theoretical results prove that the error of the STB direction is bounded by the gradient estimation error, and that a small STB update guarantees closeness to Pareto stationarity once this error is small. On five-reward fine-tuning of SD3.5-Medium, STB outperforms MGDA on all eight evaluation metrics and achieves the best results on four of the five training rewards among multi-objective diffusion RL methods.

Primary area:
generative models
Keywords:
Multi Objective, Reinforcement Learning, Diffusion Model

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_nZ3yOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper considers multi-objective preference optimization using RL fine-tuning of diffusion models, where multiple reward functions guide updates of model parameters through their gradients. These gradients can differ in magnitude and direction which can lead to a reward dominating others, motivating methods such as MGDA and MARBLE that adaptively combine them. However, the paper illustrates through a toy example that noisy gradient estimates can prevent MGDA from reaching Pareto stationarity. It also identifies highly imbalanced reward weights during diffusion RL training. To address these two issues, the paper introduces two components:

  1. Gradient smoothing: Rather than allowing noisy gradient estimates to immediately determine the update direction, the method maintains an exponential moving average of each reward’s past gradients.

  2. Anchoring: To ensure that each reward receives a minimum weight, the method incorporates a preference anchor borrowed from SDMGrad.

Critical strengths
  1. The paper is well-written and clearly structured with easy-to-follow flow.

  2. The paper is well motivated. The toy experiment and gradient analysis show how noisy gradients can affect MGDA which provides a clear motivation for the proposed method.

  3. The experiments and ablations support the paper’s main message by showing both smoothing and anchoring contribute to the improvements.

Critical weaknesses
  1. Diffusion Blend (Cheng et al., 2026) is not cited in the related work, although it falls in the same broad category as Rewarded Soup.

  2. The theory basically says that if STB’s combined direction becomes small and its gradient estimates are accurate, the model is approximately Pareto stationary. But will the direction actually become small when every objective must keep contributing because of the anchor?

  3. The weights are recomputed every HH steps, with cached weights used in between. However, the theory only analyzes the full STB direction. It is unclear how the theoretical guarantee works in between.

  4. There is no simple uniform-weight or equally weighted normalized-gradient baseline to show how much difference the proposed weighting makes.

  5. It is unclear how the preference can be incorporated in this setting.

Reference: Cheng et al (2026). Diffusion Blend: Inference-Time Multi-Preference Alignment for Diffusion Models. International Conference on Learning Representations (ICLR).

Questions for authors

In addition to the weaknesses raised above, I have the following questions:

  1. How much additional computation does STB require compared with the original DiffusionNFT training?

  2. Can user preferences be incorporated through a nonuniform preference vector pp?

  3. STB performs worse than the released DiffusionNFT checkpoint on all three held-out metrics. To better understand this comparison:

  • Did STB start from the base SD3.5-Medium model or the released DiffusionNFT checkpoint?
  • Were the same training rewards and prompt datasets used for both?
  • Were their training compute budgets matched? If so, how?
5 comments
AuthorsOpenReview ↗

Author Response

We thank the reviewer for the detailed questions on the theory, baselines, and training cost, which we address below.

W1: Missing reference to Diffusion Blend.

We will add Diffusion Blend alongside Rewarded Soups and distinguish inference-time preference alignment from STB's joint training-time optimization.

W2: Does the anchored direction actually become small?

Proposition 2 is a stationarity certificate, not a convergence proof, since an anchor can exclude a zero-norm combination available to unconstrained MGDA. Our additional analysis uses descent of the preference-weighted objective Fp=∑kpkfkF_p=\sum_kp_kf_k. With controlled estimation, normalization–anchor mismatch, and solver errors, a large direction decreases FpF_p in proportion to its squared norm and the step size, and because FpF_p is bounded below, these decreases cannot continue indefinitely. For fixed smooth objectives, positive preferences, nonzero smoothed-gradient norms, and step sizes with divergent sum, the step-size-weighted average expected squared direction norm therefore vanishes if the averaged errors vanish, and Proposition 2 then gives a vanishing average squared Pareto residual. Persistent errors leave an error neighborhood. This analyzes an idealized procedure that recomputes weights every update, not last-iterate or full AdamW convergence. We will add the full assumptions and proof to the appendix.

The mismatch arises because normalization changes the coefficients representing the preference-weighted gradient, which may violate the anchor's lower bounds. For uniform pp and λ=0.3\lambda=0.3, it is zero when the smallest smoothed-gradient norm is at least 23.1% of the mean. Within the first 1,000 steps, 100/101 recorded refreshes satisfy this condition (99.0%), and the remaining ratio is 22.8%. This supports compatibility along the recorded trajectory, not vanishing of all errors.

W3: What is guaranteed between refreshes?

We cache coefficients, not gradients. Each cached update differentiates ∑kαˉkLk\sum_k\bar\alpha_kL_k using current mini-batch gradients without objective-wise smoothing or normalization, and for fixed pp and λ\lambda the coefficient EMA preserves the anchored simplex and its coefficient floors (the cache is initialized at the feasible vector pp). A pointwise certificate also holds: by simplex feasibility and the triangle inequality, the Pareto residual is at most the norm of the cached reward-gradient combination plus the largest objective-gradient estimation error. It is informative when both terms are small, but does not prove that cached directions become small or remain optimal for the current geometry.

W4: Uniform / equally weighted normalized-gradient baseline.

The Flow-GRPO row in Table 1 is a uniform-weight baseline. It is trained on the same five rewards with fixed equal weights (each reward is normalized to unit variance and the five are summed with weight 1/5 each), and STB outperforms it on all eight metrics. To isolate the effect of the weighting within our own training setup, we also add an equally weighted normalized-gradient baseline that fixes α=p\alpha=p uniformly, the λ→∞\lambda\to\infty limit of STB, while retaining smoothing and normalization.

MethodGenEvalOCRPickScoreCLIPScoreHPSv2.1AestheticImgRwdUniRwd
Flow-GRPO (uniform weights)0.850.8722.590.2860.3095.531.403.15
λ=0\lambda=0 (no anchor)0.940.9722.740.2880.3105.681.333.25
λ=0.3\lambda=0.3 (STB)0.960.9622.910.2970.3305.701.463.29
λ=1\lambda=10.950.9622.930.2950.3315.731.493.23
Fixed uniform (λ→∞\lambda\to\infty)0.930.9122.950.2890.3255.671.413.17

The intermediate settings give similar performance with different trade-offs, rather than a unique optimum at 0.3. Fixed uniform weighting gives the highest PickScore, while STB at λ=0.3\lambda=0.3 improves the other seven metrics. This supports combining an anchor with geometry-dependent allocation.

AuthorsOpenReview ↗

Author Response continued

W5 / Q2: Can non-uniform preferences be incorporated?

Yes. For any positive preference vector pp summing to one, the effective weights are α=(w⋆+λp)/(1+λ)\alpha=(w^\star+\lambda p)/(1+\lambda) with αk≥λpk/(1+λ)\alpha_k\geq\lambda p_k/(1+\lambda), where w⋆w^\star is the geometry-dependent solution, so the anchor reserves allocation according to pp while retaining adaptive weighting. We verify this with a preference sweep that compares STB with linear scalarization (LS), which uses pp directly as fixed weights on the normalized gradients. Both continue training from STB checkpoint 450 for 80 updates under four preference vectors, ordered as GenEval / OCR / PickScore / CLIPScore / HPSv2.1: P1 = (0.450, 0.450, 0.033, 0.033, 0.033), P2 = (0.325, 0.325, 0.117, 0.117, 0.117), P3 = (0.175, 0.175, 0.217, 0.217, 0.217), and P4 = (0.050, 0.050, 0.300, 0.300, 0.300). Each entry reports STB / LS, and Start is checkpoint 450.

MetricStartP1P2P3P4
GenEval0.9260.929 / 0.9160.931 / 0.9430.932 / 0.9360.935 / 0.899
OCR0.8640.915 / 0.9350.913 / 0.9240.902 / 0.8370.871 / 0.879
PickScore22.9222.84 / 22.7522.97 / 22.9422.94 / 22.9523.03 / 23.12
CLIPScore0.2890.287 / 0.2890.290 / 0.2880.293 / 0.2900.287 / 0.293
HPSv2.10.3290.316 / 0.3080.331 / 0.3260.327 / 0.3290.332 / 0.335
Aesthetic5.815.84 / 5.735.81 / 5.785.71 / 5.765.85 / 5.83
ImgRwd1.441.38 / 1.291.45 / 1.421.41 / 1.421.44 / 1.47
UniRwd3.353.28 / 3.243.40 / 3.393.37 / 3.353.40 / 3.44

From P1 to P4, the preference shifts from GenEval and OCR toward PickScore, CLIPScore, and HPSv2.1, and STB responds with higher PickScore (22.84 → 23.03) and HPSv2.1 (0.316 → 0.332) while OCR decreases (0.915 → 0.871). The floor concerns coefficients, not guaranteed reward gains.

Q1: Additional computation relative to DiffusionNFT.

MGDA and STB recompute the balancing weights once every H=10H=10 updates. In the intervening steps, both methods reuse the most recently computed weights and perform a single backward pass on ∑kαˉkLk\sum_k \bar\alpha_k L_k, incurring the same computational cost as a weighted-sum DiffusionNFT update, including rollout generation and reward evaluation. For a fair comparison, we therefore report the average per-step update time of DiffusionNFT, MGDA, and STB over 10 consecutive updates, accounting for the periodic weight recomputation overhead. Average times per update on the same 8-GH200 hardware.

MethodAverage time per updateRelative to DiffusionNFT
DiffusionNFT (weighted sum)500.5 s1.00×
MGDA545.3 s1.09×
STB559.4 s1.12×

MGDA and STB add about 9% and 12% over DiffusionNFT, entirely from the updates that recompute the weights, and STB adds about 2.6% over MGDA. The total training time of STB is nevertheless shorter: the released DiffusionNFT model is trained for 1.7k iterations (about 236 h at the per-update time above), whereas STB uses 1,000 joint updates (about 155 h), roughly one third less.

Q3: Initialization.

STB and the controlled baselines start from base SD3.5-M with fresh LoRA, not the released DiffusionNFT checkpoint.

Q4: Training rewards and prompt datasets.

We use the same five reward types and prompt-source families (Pick-a-Pic, GenEval, OCR), but STB mixes prompts approximately 1:1:1 whereas DiffusionNFT uses a staged curriculum.

Q5: Training compute budgets.

Released DiffusionNFT is trained for 1.7k iterations versus 1,000 updates for STB, so compute is not matched and the comparison does not isolate the effect of the optimization method. We therefore compare STB against MGDA with the same backbone, data, rewards, and training budget, where STB improves all five optimized rewards and all three held-out metrics.

AuthorsOpenReview ↗

Follow-up on Reviewer Comments

Dear Reviewer nZ3y,

We have provided responses to your questions and comments. We would appreciate it if you could let us know whether you have any additional questions or concerns.

Thank you for your time and consideration.

Reviewer_A99yOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

An optimization improvement for multi-objective diffusion RL in text-to-image generation: combining per-objective gradient smoothing with a mechanism to prevent objectives from receiving persistently low weights.

Critical strengths
  1. Clearly distinguishes two issues in multi objective diffusion RL gradient estimates and persistently low weights for some objectives, and designs a dedicated mechanism for each.
  2. With the same data, rewards, and training configuration, STB outperforms MGDA on all 8 evaluation metrics.
Critical weaknesses
  1. EMA inherently reduces the angles between consecutive gradients and their fluctuations, so these diagnostics do not sufficiently demonstrate that the estimates are closer to the true gradients.
  2. The main results are limited to SD3.5-Medium, LoRA, and a fixed set of five rewards, without reporting means and variances across multiple training seeds.
Questions for authors
  1. Does gradient smoothing actually improve estimation accuracy, or does it merely reduce temporal fluctuations?
  2. Are the improvements over MGDA consistent across multiple training seeds?
  3. How sensitive is STB to the anchor strength?
5 comments
AuthorsOpenReview ↗

Author Response

We appreciate the reviewer's recognition of the motivation and controlled comparison, as well as the questions about gradient accuracy and robustness. We address these with reference-gradient probes, three-seed results, and an anchor-strength study.

W1 / Q1: Does smoothing improve accuracy, or only reduce temporal fluctuation?

Smoother gradients are not necessarily more accurate estimates. To test whether smoothing improves accuracy, rather than only reducing fluctuations, we compare raw and smoothed gradient estimates against an independent reference gradient at fixed checkpoints.

At checkpoints 300, 450, and 1,000, we sample eight batches and split them into disjoint groups of four. One group provides the reference gradient; the other provides single-, two-, and four-batch estimates. We also update the checkpoint's historical EMA with one new estimate-side batch. The table reports reference cosine alignment averaged over the five rewards.

Checkpoint1 batchMean of 2 batchesMean of 4 batchesSTB (EMA + 1 batch)
3000.1310.1800.2390.291
4500.1730.2270.2880.271
10000.1240.1540.1810.259

EMA exceeds the single-batch mean at all three checkpoints and the four-batch mean at 300 and 1,000. At 450 it is slightly below four batches (0.271 versus 0.288). This supports directional accuracy at the tested checkpoints, not uniform improvement throughout training. The finite reference remains noisy, and cosine does not measure magnitude accuracy. The smoothing-only ablation in Table 2 separately supports downstream utility.

W2 / Q2: Consistency across seeds.

Following DiffusionNFT, we use SD3.5-Medium with LoRA and five reward functions: GenEval, OCR, PickScore, CLIPScore, and HPSv2.1. This keeps the backbone and reward suite fixed when comparing the optimization methods. To assess consistency across training runs, we ran MGDA and STB with two additional seeds each, bringing the total to three seeds per method. The table reports mean ± sample standard deviation across all three runs:

MethodGenEvalOCRPickScoreCLIPScoreHPSv2.1AestheticImgRwdUniRwd
MGDA0.940 ± 0.0120.950 ± 0.00822.740 ± 0.0100.284 ± 0.0040.307 ± 0.0035.647 ± 0.0151.317 ± 0.0323.153 ± 0.051
STB (ours)0.955 ± 0.0060.960 ± 0.01022.863 ± 0.0500.296 ± 0.0060.324 ± 0.0095.747 ± 0.0571.400 ± 0.1313.267 ± 0.021

STB has higher means on all eight metrics. ImageReward has substantial variability (1.400 ± 0.131), so we do not claim statistical significance across all metrics. We will add this three-seed comparison to the experimental section of the revised manuscript and specify the training seeds in the experimental setup.

Q3: Sensitivity to the anchor strength.

We compare λ=0,0.3,1\lambda=0,0.3,1 and fixed uniform weighting under the same training budget, retaining smoothing and normalization. Zero removes the anchor; the λ→∞\lambda\to\infty limit is implemented by directly fixing α=p\alpha=p, not by passing numerical infinity. At λ=1\lambda=1, allocation remains adaptive.

λ\lambdaGenEvalOCRPickScoreCLIPScoreHPSv2.1AestheticImgRwdUniRwd
0 (no anchor)0.940.9722.740.2880.3105.681.333.25
0.3 (STB)0.960.9622.910.2970.3305.701.463.29
10.950.9622.930.2950.3315.731.493.23
Fixed uniform (λ→∞\lambda\to\infty)0.930.9122.950.2890.3255.671.413.17

The intermediate settings give similar performance with different trade-offs, rather than a unique optimum at 0.3. Removing the anchor favors OCR but lowers several general-quality rewards. Fixed uniform weighting gives the highest PickScore, while STB at λ=0.3\lambda=0.3 improves the other seven metrics. This supports combining an anchor with geometry-dependent allocation.

AuthorsOpenReview ↗

Follow-up on Reviewer Comments

Dear Reviewer A99y,

We have provided responses to your questions and comments. We would appreciate it if you could let us know whether you have any additional questions or concerns.

Thank you for your time and consideration.

Dear Reviewer,

The authors have responded to your review, and the discussion period will close soon. Please reply to let them know whether your concerns have been addressed, or which ones remain. Please also update your score or confirm that you are keeping it. Even a brief reply is very helpful.

Thank you, Area Chair

Reviewer_4TKdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors investigate the application of MGDA to multi-objective diffusion reinforcement learning and claim to have identified two problems in this setting. Since reward gradients are estimated from sampled images, they are inherently noisy, which can lead to severely imbalanced weight allocation and leave some rewards under-optimized. The authors claim that their proposed STB method addresses the first problem by smoothing each reward gradient before computing the weights, and the second by balancing the smoothed gradients using a preference anchor. The authors argue that their theoretical results demonstrate that the error of the proposed STB update direction is bounded by the gradient estimation error, and that a sufficiently small STB update guarantees that the model is close to Pareto stationarity when the estimation error is sufficiently small.

Critical strengths

The paper is clearly written and easy to read.

Critical weaknesses
  1. The paper suffers from overclaiming. In the abstract, the authors claim to have empirically identified two problems: first, reward gradients are highly noisy, making the resulting update directions unreliable; second, MGDA determines weights solely based on gradient directions, which can lead to severely imbalanced weight allocations and leave some rewards under-optimized. However, both issues have already been widely discussed in the multi-objective optimization literature. It is therefore inappropriate to present them as novel findings of this work. Similarly, I believe the authors frame well-established research challenges in the field as independent discoveries of this paper in their stated contributions.

  2. For a paper on multi-objective optimization, it is inadequate that the experiments do not include a direct evaluation of multi-objective performance, such as experiments centered on the Pareto front. Such experiments are necessary to substantiate the effectiveness of the proposed method in multi-objective optimization.

  3. As a method for multi-objective optimization, the paper should also discuss and compare against recent, advanced OPD-based methods, which have attracted considerable attention in the diffusion model community. I understand that the primary focus of this work is to improve gradient optimization. Nevertheless, comparisons with state-of-the-art methods in the field are necessary to establish the advantages of the proposed approach.

Questions for authors

Please provide a detailed explanation of why the existing experiments are sufficient to demonstrate the effectiveness of the proposed method for multi-objective optimization. In my view, most of the experiments primarily demonstrate improvements in individual objectives rather than the effectiveness of multi-objective optimization itself.

9 comments
AuthorsOpenReview ↗

Author Response

We thank the reviewer for the careful reading and the feedback on the contribution and multi-objective evaluation. Below, we clarify our research focus and provide additional preference-trade-off experiments and OPD comparisons.

W1: Novelty and overclaiming.

Our contribution concerns reliable multi-objective optimization for multi-reward diffusion RL. Recognizing gradient noise and allocation imbalance in generic multi-objective optimization (MOO) does not establish how they affect joint diffusion fine-tuning, where reward gradients are estimated from images sampled by an evolving policy. We investigate the reliability of the gradients supplied to the balancing solver and the resulting reward allocation, and develop STB to address both within a single-model training framework.

Specifically, we study how reliable the gradient geometry used to balance diffusion rewards is, and how objectives can retain a meaningful allocation when balancing produces persistently small coefficients. STB addresses the first by smoothing objective gradients before solving the MGDA problem, and the second through a preference anchor with explicit coefficient floors. We support this with current-gradient alignment probes, an estimation-to-stationarity analysis, and controlled ablations: in Table 2, smoothing alone improves all eight metrics over MGDA, anchoring improves seven and ties on OCR, and their combination is best on seven. The novelty lies in this validated approach to multi-reward diffusion optimization, not in the underlying MOO challenges, and we will revise the abstract and contributions to make this explicit.

W2: Direct multi-objective evaluation.

We evaluate whether a single diffusion model can improve multiple rewards together, rather than improving one at the expense of others. Under the same training setup, STB outperforms MGDA on all five optimized rewards and all three held-out metrics. This shows a joint improvement over the baseline, but does not tell us how the method responds to different reward preferences. We therefore add a preference-sweep experiment to examine these trade-offs directly.

We compare STB with linear scalarization (LS) at four settings of the preference vector pp in our preference-anchored balancing formulation, α=(w⋆+λp)/(1+λ)\alpha=(w^\star+\lambda p)/(1+\lambda). Here pp specifies the desired allocation across rewards, while α\alpha is the resulting adaptive allocation. LS directly uses pp as fixed weights on the normalized objective gradients. Both methods start from STB checkpoint 450 and train for 80 updates with matched gradient normalization and learning rate.

Preferences are ordered as GenEval / OCR / PickScore / CLIPScore / HPSv2.1, rounded to three decimals:

  • P1: (0.450, 0.450, 0.033, 0.033, 0.033).
  • P2: (0.325, 0.325, 0.117, 0.117, 0.117).
  • P3: (0.175, 0.175, 0.217, 0.217, 0.217).
  • P4: (0.050, 0.050, 0.300, 0.300, 0.300).

Each paired entry reports STB / LS; Start is checkpoint 450. Higher is better.

MetricStartP1P2P3P4
GenEval0.9260.929 / 0.9160.931 / 0.9430.932 / 0.9360.935 / 0.899
OCR0.8640.915 / 0.9350.913 / 0.9240.902 / 0.8370.871 / 0.879
PickScore22.9222.84 / 22.7522.97 / 22.9422.94 / 22.9523.03 / 23.12
CLIPScore0.2890.287 / 0.2890.290 / 0.2880.293 / 0.2900.287 / 0.293
HPSv2.10.3290.316 / 0.3080.331 / 0.3260.327 / 0.3290.332 / 0.335
Aesthetic5.815.84 / 5.735.81 / 5.785.71 / 5.765.85 / 5.83
ImgRwd1.441.38 / 1.291.45 / 1.421.41 / 1.421.44 / 1.47
UniRwd3.353.28 / 3.243.40 / 3.393.37 / 3.353.40 / 3.44

Moving from P1 to P4 places less emphasis on GenEval and OCR and more on PickScore, CLIPScore, and HPSv2.1. STB responds with higher PickScore (22.84 → 23.03) and HPSv2.1 (0.316 → 0.332), while OCR decreases (0.915 → 0.871). This shows that changing pp can steer the model toward different reward trade-offs. Compared with LS, neither method is best on every reward: at P4, for example, LS achieves higher PickScore (23.12 versus 23.03), while STB retains higher GenEval (0.935 versus 0.899). The experiment therefore evaluates how the methods balance competing rewards, rather than only whether individual scores improve.

AuthorsOpenReview ↗

Author Response continued

W3: Comparison with OPD-based methods.

We evaluated released DiffusionOPD and DreOPD SD3.5-M students using our evaluation pipeline and their recommended CFG setting:

Model (CFG)GenEvalOCRPickScoreCLIPScoreHPSv2.1AestheticImgRwdUniRwd
DiffusionOPD (4.5)0.960.9424.010.2970.3426.151.513.57
DreOPD (4.5)0.970.9324.030.2950.3496.241.533.60
STB (ours, 1.0)0.960.9622.910.2970.3305.701.463.29

The OPD students are stronger on PickScore, HPSv2.1, and the held-out metrics; STB is stronger on OCR and comparable on GenEval and CLIPScore.

Different training problems. These multi-teacher OPD methods separate learning individual capabilities from combining them: reward-specialized teachers provide supervision for a unified student on its own rollouts. STB instead optimizes all five rewards directly in one joint RL run from the base diffusion model, without pretrained reward-specialist teachers. Our method addresses how to learn and balance these capabilities jointly, rather than how to consolidate capabilities already acquired by teachers.

Different training and inference costs. OPD first trains reward-specialist teachers through separate RL runs and then distills them into a student, whereas STB is a single joint RL run from the base model. The released OPD students also rely on classifier-free guidance (without it, their GenEval drops to 0.15 and 0.41 in our evaluation), so each denoising step needs a conditional and an unconditional forward pass. Generating one image with 40 denoising steps therefore takes 80 forward passes of the diffusion model instead of the 40 needed by STB. OPD times are from Table 1 of DiffusionOPD, and the STB time is measured from our training logs. Since DiffusionOPD does not report its training hardware, the wall-clock times are given for reference only.

MethodTraining pipelineWall-clock timeHardwareForward passes per generated image
DiffusionOPD3 teacher RL runs + distillation46.92 + 33.17 + 85.75 + 11.26 = 177.10 hnot reported40 steps × 2 = 80 (CFG 4.5)
STB (ours)1 joint RL run1,000 × 559.4 s = 155.4 h8 GH20040 steps × 1 = 40 (no CFG)

Q1: Why the existing experiments demonstrate multi-objective effectiveness.

Improving individual objectives would appear as gains on some rewards paid for by losses on others, and Table 2 shows the opposite. All rows train one model on the same five rewards with the same data and training configuration, and STB is higher than MGDA on every optimized reward (GenEval 0.96 vs. 0.93, OCR 0.96 vs. 0.95, PickScore 22.91 vs. 22.73, CLIPScore 0.297 vs. 0.283, HPSv2.1 0.330 vs. 0.305) and on all three held-out metrics. No objective is traded away, so the STB solution Pareto-dominates the MGDA solution, which is the standard criterion for comparing two multi-objective solutions. Since STB differs from MGDA only in how the objectives are combined, these gains come from better multi-objective balancing, and the preference sweep in W2 further shows how STB moves along the trade-off surface.

References

  1. Li, Quanhao, et al. "DiffusionOPD: A unified perspective of on-policy distillation in diffusion models." arXiv preprint arXiv:2605.15055 (2026).
  2. Lin, Mingfeng, et al. "Dreopd: Degraded-reference extrapolative on-policy distillation for flow-matching models." arXiv preprint arXiv:2608.09233 (2026).
AuthorsOpenReview ↗

Follow-up on Reviewer Comments

Dear Reviewer 4TKd,

We have provided responses to your questions and comments. We would appreciate it if you could let us know whether you have any additional questions or concerns.

Thank you for your time and consideration.

Comments on the paper

2 comments

Dear Reviewers,

The authors have posted their responses. Could you please read the rebuttal and let us know whether it resolves the questions and concerns raised in your review? If any point remains unclear, feel free to follow up with the authors during the discussion period.

Please update your review and final rating at your earliest convenience.

Thank you very much for your time and contribution.

Best regards, AC

AuthorsOpenReview ↗

Rebuttal Summary

Dear Area Chair and Reviewers,

We thank all reviewers for the constructive feedback and the Area Chair for facilitating the discussion. The reviewers' main concerns focused on STB's novelty, multi-objective effectiveness, gradient accuracy, robustness, and theoretical guarantees. We addressed these through additional preference-sweep, reference-gradient, multi-seed, and baseline experiments, along with theoretical clarifications. We also clarified the differences between STB and OPD methods, highlighting that STB achieves competitive performance through a single joint RL run without requiring pretrained reward-specialist teachers. We appreciate Reviewer nZ3y's positive feedback, acknowledging that our responses resolved the empirical concerns and raising the rating. Although the other two reviewers have not yet responded, we hope our additional results and clarifications have helped address the main concerns. We sincerely appreciate the reviewers' and Area Chair's time and consideration.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.