- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes to use a pretrained T2I model as the backbone for a reward model. Image features are pooled from selected MMDiT layers and mapped to a scalar reward and trained with Bradley-Terry objective. Using the HPSv3 preference data mixture, the SD3.5-Large and Qwen-Image variants are reported to beat HPSv3 on four preference benchmarks. the model is also used as a Flow-GRPO reward signal, and a shared latent space with the policy enables a 1.65x inference speedup.
Critical strengths
- The layer-wise probing on a frozen backbone is a well-supported contribution. The head with ~10M reaches 72.3% on HPSv3 without adapting the generator. This substantiates the central claim that generative representations already encode preference-relevant structure.
- The prompt-alignment regression under RL, reward hacking in both reward models, and the position effect in the human study are all surfaced rather than hidden.
Critical weaknesses
- The paper does not distinguish it from the closest prior work. LRM [1] already proposed using the diffusion model itself as a latent reward model, yet this paper cites it only in a group citation and reduces it to a baseline in Table 1, while framing "can generative representations support evaluation" as if the paradigm were new.
- The method hinges on scoring a near-clean latent, yet is asserted as a fixed constant with no sensitivity analysis. [1] Diffusion Model as a Noise-Aware Latent Reward Model for Step-Level Preference Optimization. Tao Zhang, Cheng Da, et al.
Questions for authors
- What is the essential difference between this method and the earlier diffusion-based LRMs? How do they differ as RL reward signals, and how do their inference latencies compare?
- The RL evaluation uses only the SD3.5-L reward with an SD3.5 policy, which shares a generative prior with the policy. Does the stronger Qwen-Image (Heavy) reward, a different backbone from the policy, still win under RL?
- The method hinges on scoring a near-clean latent, yet is asserted as a fixed constant with no sensitivity analysis.
5 comments
Key Differences Between DiT-Reward and LRM: Part2
3. From Feasibility to High-Accuracy and Practically Effective Reward Modeling
We emphasize that reward prediction accuracy is far more than a secondary benchmark metric. Generative reinforcement learning is highly sensitive to reward quality and susceptible to reward hacking. Therefore, accurate preference prediction is essential for providing meaningful optimization signals.
A model that produces approximately reasonable reward scores and a model that can reliably guide generative optimization have fundamentally different practical value.
Under the same public preference-training data mixture as HPSv3, DiT-Reward outperforms HPSv3 across all four evaluated preference benchmarks. In particular, our Qwen-Image variant achieves 79.1% accuracy on HPDv3, compared with 76.9% for HPSv3, representing a 2.2-percentage-point improvement.
This demonstrates that modern generative backbones can serve not merely as feasible reward representations, but as competitive alternatives to strong VLM-based reward models.
More importantly, our contribution extends beyond offline benchmark improvements. We further investigate whether reward-model quality translates into improvements in generated images.
In a matched Flow-GRPO experiment, the policy optimized with DiT-Reward achieves a 75.1% win rate in an independent human evaluation involving 2,048 image pairs, substantially outperforming the policy optimized with HPSv3.
While LRM demonstrated that diffusion models can provide reward signals, our work further shows that modern generative representations can support highly competitive reward modeling and translate this capability into substantial improvements in human-preferred generation.
We believe that the transition from feasibility to high-accuracy reward modeling is significant. Only when generative reward models achieve sufficiently competitive discriminative performance can they become viable candidates for widespread adoption in practical reinforcement learning systems.
4. Differences in Research Objectives: Step-Wise Rewards vs. High-Accuracy Final-Image Evaluation
LRM primarily focuses on learning noise-aware reward signals across diffusion timesteps to support Step-wise Latent Preference Optimization (LPO). Its design therefore emphasizes providing usable reward feedback at different intermediate noisy states.
In contrast, DiT-Reward focuses on high-accuracy preference evaluation of completed images and operates on near-clean latents.
Our noise-sensitivity experiments further indicate that reward prediction accuracy decreases as the input noise level increases. We will provide the corresponding ablation results in our supplementary experiments to further justify this design choice.
Therefore, the two works emphasize different objectives: LRM focuses on reward modeling across intermediate diffusion states for step-wise optimization, whereas DiT-Reward focuses on extracting the preference-discriminative capability of generatively pretrained representations to evaluate completed images and guide downstream RL.
Summary
We believe DiT-Reward contributes three mutually reinforcing advances beyond LRM:
- Architecture and Scaling: Systematically investigating reward representations in modern large-scale MMDiTs and providing evidence of positive reward scaling with stronger generative backbones.
- Scientific Understanding: Revealing that pretrained generative representations already encode transferable preference information, and characterizing its organization and complementarity across Transformer depths.
- Practical Value: Advancing diffusion-native reward modeling to competitive accuracy against strong VLM-based reward models, with effectiveness further validated through independent human evaluation.
If LRM established the feasibility of using diffusion models as reward models, DiT-Reward further demonstrates the evaluative potential, scalability, and practical value of modern large-scale generative Transformers.
Noise-Level τ Ablation
Thank you for suggesting a noise-level ablation. Following your recommendation, we conducted additional experiments with τ = 0.0, 0.1, 0.5, and 0.8, keeping the backbone, training data, feature layers, and optimization settings unchanged. We include the paper’s original τ = 0.005 setting for reference.
Table. Reward prediction accuracy (%) across noise levels.
| τ | HPDv3 ↑ | ImageReward ↑ | HPDv2 ↑ | PickScore (Pick-a-Pic) ↑ |
|---|---|---|---|---|
| 0.0 | 77.06 | 66.39 | 86.08 | 65.50 |
| 0.005 (original) | 77.60 | 67.00 | 85.60 | 66.70 |
| 0.1 | 77.56 | 67.15 | 85.71 | 67.20 |
| 0.5 | 76.22 | 64.99 | 85.28 | 66.60 |
| 0.8 | 72.21 | 61.73 | 77.96 | 64.10 |
Small perturbations can benefit reward prediction, with performance varying only modestly within the low-noise regime. Larger τ reduces the input signal-to-noise ratio and is associated with lower accuracy. Notably, τ = 0.1 modestly improves three of the four benchmarks over our original setting while maintaining comparable HPDv3 performance; we will consider this configuration in future experiments. We thank the reviewer for motivating this informative ablation.
Key Differences Between DiT-Reward and LRM: Part1
We thank the reviewer for raising this important question. We fully acknowledge LRM's pioneering contribution to diffusion-native reward modeling, which we have already cited in our paper. We do not claim to be the first to use diffusion models as reward models. Building on this foundation, DiT-Reward investigates the preference representation capabilities, scaling behavior, and practical value of modern large-scale generative Transformers as high-accuracy reward models.
1. Model Architecture and Scaling: Beyond Simply Replacing the Backbone
LRM primarily investigates U-Net-based diffusion architectures, adopting a CLIP-style reward formulation based on cosine similarity between projected visual and textual features, together with Visual Feature Enhancement (VFE). Although LRM also includes preliminary MMDiT experiments, it does not systematically investigate reward representations or scaling behavior in large-scale MMDiTs.
In contrast, DiT-Reward is specifically designed around MMDiT. Rather than explicitly computing text-image cosine similarity, we directly extract preference-relevant representations from image tokens that have already interacted with textual information through joint attention. Our downstream human evaluation demonstrates that this formulation can achieve highly competitive outcomes against VLM-based rewards.
Importantly, our contribution is not merely adopting a different backbone. We systematically investigate how Transformer depth, layer selection, and cross-layer feature aggregation affect reward prediction. These experiments provide insights into how reward-relevant information can be effectively extracted from large-scale generative Transformers, which were not systematically established in LRM.
More importantly, we investigate whether reward modeling capability improves as the generative backbone scales. Across SD3.5-Medium, SD3.5-Large, and Qwen-Image, we observe positive reward scaling trends. The within-family SD3.5 comparison provides more controlled evidence, while Qwen-Image extends our observations across model families.
To our knowledge, our work is among the first to explicitly investigate and demonstrate positive reward scaling in diffusion-native reward modeling. This finding suggests a promising direction for developing stronger reward models by leveraging advances in large-scale generative pretraining.
Scaling generative Transformers can benefit not only image generation but also preference discrimination. This relationship was not systematically established in LRM.
2. Scientific Finding: Native DiT Representations Are Transferable to Reward-Based Discriminative Tasks
An important scientific contribution of DiT-Reward is the finding that generative Transformers learn not only how to generate images, but also internal representations containing discriminative information useful for evaluating them. Knowledge acquired through generative pretraining is therefore not limited to generation, but can transfer to downstream tasks such as human preference modeling.
More specifically, native intermediate DiT representations already encode high-level semantic information from which rich preference-relevant signals can be extracted without fine-tuning the generative backbone. This suggests that reward modeling need not learn an entirely separate visual evaluation representation from scratch.
Our experiments support this perspective. First, freezing the entire generative backbone and training only a lightweight reward head with approximately 10M parameters achieves 72.3% accuracy on HPDv3. Second, layer-wise probing reveals that preference information is not uniformly distributed across network depth, with particularly strong predictive representations emerging in middle-to-late layers. Third, cross-layer analysis identifies complementary information across different stages, demonstrating the distinct discriminative value of multi-stage generative representations.
These findings advance our understanding from "generative models can be adapted into reward models" to "generative models already acquire transferable evaluative representations during pretraining."
This insight motivates our architectural choices and suggests a broader research direction: powerful generative models may serve not only as foundation models for generation, but also as general-purpose representation backbones for visual evaluation and preference learning.