acceptodds
Under review as a conference paper at ICLR 2027

DualHeads: Reward-tilted few-step diffusion made simple

Abstract

Distilled diffusion models generate high-quality images in a few sampling steps, but aligning them with task requirements and human preferences is difficult. Methods developed for multi-step diffusion models rely on differentiable rewards, or on score fields and stochastic trajectories that distilled generators may lack. We ask whether a few-step generator can be adapted using only external reward evaluations and reference samples, and show that two binary classifiers suffice. Our method, DualHeads, implements them as lightweight heads on a frozen diffusion backbone. The reward head learns which samples of the same prompt score higher, the reference head separates reference images from generated ones, and the generator maximizes a weighted sum of their logits. When the heads are optimal, this objective is KL-regularized maximization of the learned reward, and the ratio of head weights sets the regularization strength. Empirically, with both rule-based and model-based rewards, a few-step DualHeads generator reaches task rewards close to those of 40-step reinforcement learning methods while preserving image quality, at no extra inference cost. We hope that this simple and elegant dual-head formulation will help advance few-step reward-tilted generation.

Primary area:
generative models
Keywords:
diffusion models, diffusion distillation, diffusion reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_9mEyOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

DualHeads adapts few-step diffusion generators using two classifiers on a frozen backbone: one learns a reward surrogate, and the other encourages agreement with reference images. The method supports black-box rewards and adds no inference overhead. Results across OCR, GenEval, and preference rewards are promising.

Critical strengths
  1. The approach combines black-box reward optimization and reference matching without requiring trajectory likelihoods or an online generator-score model.

  2. Experiments span multiple rewards and two backbones, with held-out quality metrics providing evidence beyond improvement on the training reward.

Critical weaknesses
  1. Insufficient controls to establish the contribution. The OCR reference pool is already reward-filtered, so reference matching alone could explain part of the gain. Missing controls include reference-only training, reward-only training, and freezing the reward head after warm-up. A simple alternative worth comparing is supervised fine-tuning on the same high-reward reference images. These comparisons would establish whether online dual-head training is necessary.

  2. Claims need tighter qualification. The KL interpretation applies to idealized classifiers and noised, recomputed predictions; its connection to the final sampler distribution and detached training update is not established. Patchwise classification on frozen features further restricts the density-ratio interpretation. Similarly, improved preference scores do not establish diversity preservation, and the reference-pool ablation does not support a clean separation between reward and reference-head responsibilities.

Questions for authors
  1. How does full DualHeads compare with reference-only training and a frozen, warmed-up reward head under identical conditions?

  2. Which claims apply exactly to the implemented update, and which are idealized interpretations? What connects the recomputed-prediction objective to final generated samples?

  3. What is the exact initial SD3.5-M distilled checkpoint, and how does it perform before adaptation? Can Table 2 comparisons use matched resolutions?

  4. What are the total GPU-hours, peak memory, reward-query count, and reference-pool construction cost? The heads contain 92.5M–208.3M parameters and require gradients through a frozen backbone, so practical training efficiency remains unclear.

Suggested Improvements:

  1. Add the component controls above and one simple reference-data fine-tuning baseline.

  2. Report task reward, held-out quality, and sample diversity together across the reward-weight sweep.

  3. Clarify evaluation uncertainty and distinguish prompt-level variation from variation across training runs.

  4. Position the contribution as a practical simplification; the reported results do not establish consistent superiority over existing few-step methods.

Reviewer_f44zOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose DualHeads, a method that they claim adapts few-step diffusion generators using only external reward evaluations and reference samples. The authors claim that their method transforms KL-regularized reward maximization into two binary classification problems, which are addressed using lightweight classifier heads built on a frozen diffusion backbone.

Critical strengths

The paper is clearly written and well-formatted.

Critical weaknesses
  1. The proposed method is relatively simple, mainly introducing two binary classification heads on a frozen diffusion backbone to achieve reward optimization and reference distribution regularization. However, the authors have not sufficiently clarified the fundamental differences between this design and existing methods based on classifier guidance, reward modeling, and classification-based distribution matching. I suggest that the authors further clarify the specific innovations of the two classification heads and their optimization mechanisms compared with prior work, and provide stronger justification for why these designs are necessary and cannot be easily replaced by existing approaches.

  2. The authors repeatedly emphasize the computational efficiency advantages of their method, particularly few-step generation and zero additional inference overhead. However, the paper lacks a quantitative analysis of the additional computational costs introduced by the two classification heads during training. Therefore, I believe that the authors have not sufficiently validated their claims regarding computational efficiency, and the experimental evidence remains incomplete.

  3. Given that the paper emphasizes theoretical derivations as one of its main contributions, the mathematical notation should be presented more rigorously. However, throughout the paper, mathematical symbols and equations do not consistently use appropriate font styles to distinguish variables according to their mathematical meanings and properties.

  4. The workflow diagram in Fig. 2 is too small to read clearly. I suggest that the authors enlarge the figure to improve its readability.

  5. The proposed method is overly simple, and the authors need to clearly explain how the two binary classifiers differ from those used in existing methods.

Questions for authors
  1. Reinforcement learning with black-box rewards relies on certain properties that are lost during the distillation process. The authors argue that policy gradients require stochastic multi-step sampling trajectories to estimate update directions, and that these estimates exhibit high variance when the number of sampling steps is reduced to only a few. How does the proposed method ensure that it does not suffer from similarly high variance? I did not find any analysis or experiments addressing this issue in the paper.

  2. The authors introduce two additional classifier networks. What are the sizes of these two networks, and what computational overhead do they introduce?

  3. Since both classification heads are trained alongside the main diffusion model, they may be unstable at the beginning of training. Could this instability affect the overall generation quality during the initial training stage?

Given the overall quality of the paper, I am willing to consider raising my score if the authors adequately address my concerns.

2 comments

Dear Reviewer,

The authors have provided their responses to the raised concerns and revisions. Please review their responses and join the discussion before the end of the author-review discussion phase (Nov 18, AoE). Thanks.

Best, AC

Dear AC cidG,

Thank you for your reminder. I have provided my responses and concerns for your review. Thank you again for your work.

Reviewer_u6PFOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

Final Justification after Rebuttal

Firstly, I appreciate the authors' efforts in the rebuttal. After reading the other reviewer's comments, I agree with their concerns about this work.

However, I still find the practical applicability of this work meaningful, particularly given its simple intervention in Diffusion models pipeline. Thus, I would like to maintain my supportive score for this work.


This paper focuses on a quite intuitive problem formulation. Since a distilled generative model (e.g., Yin et al., 2023) is a black box (e.g., we all know that a formal diffusion model follows an objective like L=E∥vθ(xt,t)−(x1−ϵ)∥2\mathcal L = \mathbb E \| v_\theta (x_t,t) - (x_1 - \epsilon) \|^2; however, once it is distilled, we don't know the exact objective), adopting a simple additional reward (like Zheng et al., 2025) could even make the model worse (Table 4).

So, I think the work focuses just on 'generating good samples' itself: whatever 'score' is finally constructed (e.g., however the model is distilled (Huang et al., 2026; Luo et al., 2026)), it treats the models as just black boxes and then applies a generative signal that relies only on the samples, so that this framework is applicable to arbitrary distilled generative models.

To satisfy the two goals (sufficiently close to the reference distribution w/ high reward), they leverage formal RLFH objective function Jc(θ)=Epθ[log⁡pθ−log⁡pref−rβ]=KL(pθ∣p∗)−log⁡Z\mathcal{J} _c(\theta) = \mathbb{E} _{p _\theta}\Big[\log p _\theta - \log p _{\text{ref}} - \tfrac{r}{\beta}\Big]= \mathrm{KL} (p _\theta \mid p^*) - \log Z. Here, the work proposes the additional modules called the dual head (the reference head DϕD _\phi and the reward head RψR _\psi) to optimize J\mathcal J.

Critical strengths

Q1. Problem formulation
I think this kind of concept, which mitigates per-model RL optimization, is quite intuitive and reasonable. Additionally, the novelty claim is also well established (except for line 67, which claims that Existing methods fall into two families, each one addressing typically only one side of this setting.).

Q2. Experiments are well-designed for supporting the main claim
This work successfully shows that, w/ samples only, it is capable of designing a proper reward. Specifically, on SD3.5-M, the four-step DualHeads improves GenEval from 0.63 to 0.90 and OCR from 0.59 to 0.92 while improving all held-out metrics over the base model (Table 1), and it also works w/ a model-based reward (HPSv3, 11.23 → 13.93 on Z-Image-Turbo, Table 2). Moreover, the βeq\beta_{\text{eq}} sweep (Fig. 4a) shows that only the ratio λR/λD\lambda_R / \lambda_D matters, which directly supports the theoretical claim.

Critical weaknesses

Please refer to the Questions section below. Most of my concerns are requests for clarification or additional evidence, rather than critical weaknesses that undermine the main contributions.

Questions for authors

Q1. Comparison with REST (Li et al., 2026)
I fully understand that this is concurrent work. However, since DualHeads is capable of handling arbitrary reward functions and even shows the ability to handle multiple rewards at once (Table 4, where J=PickScore + CLIPScore + HPSv2.1\mathcal J = \text{PickScore + CLIPScore + HPSv2.1}), I think matching the exact setup of Li et al., 2026 (or at least matching the NFEs, either by setting DualHeads to 8 steps or REST to 2 or 4 steps) with a PickScore-reinforced version of DualHeads would be helpful for understanding the overall work.

Q2. Would additional experiments be possible?
The main weaknesses of this submission seem to stem largely from the limited experiments. In particular, I would appreciate the following, even if only a subset is feasible:

  • Per-head ablations. Runs with λR=0\lambda_R = 0 (reference head only) and λD=0\lambda_D = 0 (reward head only). Since the OCR reference pool is filtered by the OCR reward (best-of-8, score ≥0.6\geq 0.6), part of the OCR gain may come from the reference head alone; a reference-head-only run with an unfiltered pool would further disentangle the contributions. The λD=0\lambda_D = 0 run would also directly support the claim that the reference head preserves image quality.
  • Initial student. Please specify which distilled SD3.5-M checkpoint is used for initialization and report its scores in Table 1, so that the gain from DualHeads can be separated from that of distillation.
  • Generality. At least one additional few-step generator obtained by a different distillation method (e.g., consistency-based) would support the claim of applicability to arbitrary few-step generators.

Minor. Please fix line 67 and related context. I don't think this is typical, since the authors also note in Section 2 that Luo et al., 2026 already handles both sides (i.e., few-step generators and non-differentiable rewards).


Overall, this paper has weaknesses mainly related to the experimental budget. However, I think these kinds of weaknesses do not harm the paper's main claim; in particular, the main claim is well supported, and the initial intuition seems sufficient for further discussion. Therefore, I recommend clear acceptance of this submission.

Reviewer_hbHJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes DualHeads for post-training few-step diffusion generators using black-box rewards and prompt-paired reference images. Two classifiers share a frozen diffusion backbone: a reward head learns soft labels derived from group-relative rewards, while a reference head distinguishes reference images from recomputed generator predictions. The generator is updated through a weighted sum of their logits. Under idealized classifier assumptions, the authors derive a reverse-KL interpretation for the noised prediction distribution.

Experiments cover OCR, GenEval, and preference rewards on SD3.5-M and Z-Image. The reported four-step models achieve OCR and GenEval scores of 0.92 and 0.90, with improvements on several held-out preference metrics.

Critical strengths
  1. The method offers a useful simplification of few-step reward optimization. The proposed post-training loop supports non-differentiable rewards without requiring trajectory likelihoods, external reward gradients, or online teacher/fake-score fitting. Both heads use a common classification formulation and are discarded at inference. Reducing these modeling requirements is a practically meaningful contribution.

  2. The experiments demonstrate competitive performance across different settings. The evaluation covers rule-based and learned rewards, two generator families, and both two- and four-step inference. Strong OCR and GenEval results, together with improvements on several held-out metrics, support the feasibility of this approach. The reference-pool and warm-up experiments also provide useful information about its behavior.

Critical weaknesses
  1. The experiments do not sufficiently isolate the contribution of the proposed reward-learning mechanism. For OCR, the reference images already incorporate substantial task supervision: the authors select the best of eight candidates and retain it only above a reward threshold of 0.6 (lines 321–323). However, there is no reference-head-only control using the same selected pool. Consequently, the current results do not establish how much improvement requires the reward head beyond adversarial adaptation to reward-curated references. Furthermore, the warm-up ablation compares two reward heads that both continue learning online. It therefore does not isolate the benefit of online refitting, an important distinction from approaches using a fixed reward head. Since the contribution centers on a simpler online surrogate formulation, these missing controls materially limit the empirical attribution.

  2. The principal comparisons do not fully establish the gains under matched conditions. Table 2 compares DualHeads at the paper’s stated 5122512^2 setting with Z-Image-Turbo and TDM-R1 at 102421024^2 for the model-based metrics. Only the daggered baseline OCR and GenEval entries use 5122512^2. This disclosed resolution difference prevents the HPSv3 comparison from cleanly isolating the training method’s effect. Separately, the SD3.5-M experiments do not clearly identify the exact distilled initialization and report its complete pre-training metrics under the same evaluation protocol. Table 5 includes a four-step TDM-SD3.5-M result, but does not explicitly connect that checkpoint to the initialization used. Comparing primarily against the multi-step base model leaves the improvement over the actual starting student insufficiently characterized. These issues matter because competitive performance with fewer modeling requirements is the main empirical justification for the method.

Questions for authors
  1. Component attribution: With identical initialization, reference data, and training budget, how does DualHeads compare with reference-head-only training? How does keeping a warmed-up reward head frozen compare with refitting it online? A focused comparison on one task would directly address the first weakness.

  2. Matched evaluation: Can the key Table 2 comparisons be reported at a common 5122512^2 resolution using the same prompts, preprocessing, and evaluation protocol? Please also specify evaluation sample counts and the uncertainty of the reported differences.

  3. Initialization and warm-up: What exact distilled checkpoint initializes the SD3.5-M experiments, and what are its complete starting metrics? During reward-head warm-up, how are informative group-relative labels constructed when each prompt has at most one selected reference image? Please clarify whether additional candidates or a different labeling procedure are used.

  4. Theoretical scope: I understand Eqs. (10)–(11) as an idealized characterization of the noised, recomputed prediction distribution, as noted in the paper. What additional assumption or empirical evidence connects this surrogate to the deployed few-step sampler? This clarification would help distinguish the established objective identity from the interpretation of the implemented update.

1 comment

Dear Reviewer,

The authors have provided their responses to the raised concerns and revisions. Please review their responses and join the discussion before the end of the author-review discussion phase (Nov 18, AoE). Thanks.

Best, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.