acceptodds
Under review as a conference paper at ICLR 2027

DiT-Reward: Generative Representations for Text-to-Image Reward Modeling

Abstract

Can representations learned for image generation also support the evaluation of generated images? We study text-to-image reward prediction as a downstream task of generative representation learning and introduce DiT-Reward, which repurposes pretrained diffusion transformers by aggregating image representations conditioned on text across layers. Probing a frozen SD3.5-Large backbone reveals that preference information is already accessible through a lightweight learned head, with stronger predictive features in the middle and later layers and complementary information across stages. Using the same public preference data mixture as HPSv3, both our SD3.5-Large and Qwen-Image variants outperform HPSv3 on all four evaluated benchmarks, with the Qwen-Image variant reaching 79.1% accuracy on HPDv3. Backbone comparisons further indicate a positive scaling trend with generative backbone capacity. When used as the reward for Flow-GRPO, the SD3.5-Large variant produces a policy that wins 75.1% of 2,048 human comparisons against the policy optimized with HPSv3, with 21.6% losses and 3.3% ties, after aggregating five ratings per image pair. With a shared latent space between the policy and reward model, direct latent scoring also achieves a 1.65× reward inference speedup over HPSv3 with comparable peak memory. These results establish generative representations as an effective foundation for reward modeling and policy optimization.

Primary area:
generative models
Keywords:
text-to-image generation, reward modeling, diffusion transformers, generative representation learning, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_hHpSOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes to use a pretrained T2I model as the backbone for a reward model. Image features are pooled from selected MMDiT layers and mapped to a scalar reward and trained with Bradley-Terry objective. Using the HPSv3 preference data mixture, the SD3.5-Large and Qwen-Image variants are reported to beat HPSv3 on four preference benchmarks. the model is also used as a Flow-GRPO reward signal, and a shared latent space with the policy enables a 1.65x inference speedup.

Critical strengths
  1. The layer-wise probing on a frozen backbone is a well-supported contribution. The head with ~10M reaches 72.3% on HPSv3 without adapting the generator. This substantiates the central claim that generative representations already encode preference-relevant structure.
  2. The prompt-alignment regression under RL, reward hacking in both reward models, and the position effect in the human study are all surfaced rather than hidden.
Critical weaknesses
  1. The paper does not distinguish it from the closest prior work. LRM [1] already proposed using the diffusion model itself as a latent reward model, yet this paper cites it only in a group citation and reduces it to a baseline in Table 1, while framing "can generative representations support evaluation" as if the paradigm were new.
  2. The method hinges on scoring a near-clean latent, yet τ=0.005\tau=0.005 is asserted as a fixed constant with no sensitivity analysis. [1] Diffusion Model as a Noise-Aware Latent Reward Model for Step-Level Preference Optimization. Tao Zhang, Cheng Da, et al.
Questions for authors
  1. What is the essential difference between this method and the earlier diffusion-based LRMs? How do they differ as RL reward signals, and how do their inference latencies compare?
  2. The RL evaluation uses only the SD3.5-L reward with an SD3.5 policy, which shares a generative prior with the policy. Does the stronger Qwen-Image (Heavy) reward, a different backbone from the policy, still win under RL?
  3. The method hinges on scoring a near-clean latent, yet τ=0.005\tau=0.005 is asserted as a fixed constant with no sensitivity analysis.
5 comments
AuthorsOpenReview ↗

Key Differences Between DiT-Reward and LRM: Part1

We thank the reviewer for raising this important question. We fully acknowledge LRM's pioneering contribution to diffusion-native reward modeling, which we have already cited in our paper. We do not claim to be the first to use diffusion models as reward models. Building on this foundation, DiT-Reward investigates the preference representation capabilities, scaling behavior, and practical value of modern large-scale generative Transformers as high-accuracy reward models.

1. Model Architecture and Scaling: Beyond Simply Replacing the Backbone

LRM primarily investigates U-Net-based diffusion architectures, adopting a CLIP-style reward formulation based on cosine similarity between projected visual and textual features, together with Visual Feature Enhancement (VFE). Although LRM also includes preliminary MMDiT experiments, it does not systematically investigate reward representations or scaling behavior in large-scale MMDiTs.

In contrast, DiT-Reward is specifically designed around MMDiT. Rather than explicitly computing text-image cosine similarity, we directly extract preference-relevant representations from image tokens that have already interacted with textual information through joint attention. Our downstream human evaluation demonstrates that this formulation can achieve highly competitive outcomes against VLM-based rewards.

Importantly, our contribution is not merely adopting a different backbone. We systematically investigate how Transformer depth, layer selection, and cross-layer feature aggregation affect reward prediction. These experiments provide insights into how reward-relevant information can be effectively extracted from large-scale generative Transformers, which were not systematically established in LRM.

More importantly, we investigate whether reward modeling capability improves as the generative backbone scales. Across SD3.5-Medium, SD3.5-Large, and Qwen-Image, we observe positive reward scaling trends. The within-family SD3.5 comparison provides more controlled evidence, while Qwen-Image extends our observations across model families.

To our knowledge, our work is among the first to explicitly investigate and demonstrate positive reward scaling in diffusion-native reward modeling. This finding suggests a promising direction for developing stronger reward models by leveraging advances in large-scale generative pretraining.

Scaling generative Transformers can benefit not only image generation but also preference discrimination. This relationship was not systematically established in LRM.

2. Scientific Finding: Native DiT Representations Are Transferable to Reward-Based Discriminative Tasks

An important scientific contribution of DiT-Reward is the finding that generative Transformers learn not only how to generate images, but also internal representations containing discriminative information useful for evaluating them. Knowledge acquired through generative pretraining is therefore not limited to generation, but can transfer to downstream tasks such as human preference modeling.

More specifically, native intermediate DiT representations already encode high-level semantic information from which rich preference-relevant signals can be extracted without fine-tuning the generative backbone. This suggests that reward modeling need not learn an entirely separate visual evaluation representation from scratch.

Our experiments support this perspective. First, freezing the entire generative backbone and training only a lightweight reward head with approximately 10M parameters achieves 72.3% accuracy on HPDv3. Second, layer-wise probing reveals that preference information is not uniformly distributed across network depth, with particularly strong predictive representations emerging in middle-to-late layers. Third, cross-layer analysis identifies complementary information across different stages, demonstrating the distinct discriminative value of multi-stage generative representations.

These findings advance our understanding from "generative models can be adapted into reward models" to "generative models already acquire transferable evaluative representations during pretraining."

This insight motivates our architectural choices and suggests a broader research direction: powerful generative models may serve not only as foundation models for generation, but also as general-purpose representation backbones for visual evaluation and preference learning.

AuthorsOpenReview ↗

Key Differences Between DiT-Reward and LRM: Part2

3. From Feasibility to High-Accuracy and Practically Effective Reward Modeling

We emphasize that reward prediction accuracy is far more than a secondary benchmark metric. Generative reinforcement learning is highly sensitive to reward quality and susceptible to reward hacking. Therefore, accurate preference prediction is essential for providing meaningful optimization signals.

A model that produces approximately reasonable reward scores and a model that can reliably guide generative optimization have fundamentally different practical value.

Under the same public preference-training data mixture as HPSv3, DiT-Reward outperforms HPSv3 across all four evaluated preference benchmarks. In particular, our Qwen-Image variant achieves 79.1% accuracy on HPDv3, compared with 76.9% for HPSv3, representing a 2.2-percentage-point improvement.

This demonstrates that modern generative backbones can serve not merely as feasible reward representations, but as competitive alternatives to strong VLM-based reward models.

More importantly, our contribution extends beyond offline benchmark improvements. We further investigate whether reward-model quality translates into improvements in generated images.

In a matched Flow-GRPO experiment, the policy optimized with DiT-Reward achieves a 75.1% win rate in an independent human evaluation involving 2,048 image pairs, substantially outperforming the policy optimized with HPSv3.

While LRM demonstrated that diffusion models can provide reward signals, our work further shows that modern generative representations can support highly competitive reward modeling and translate this capability into substantial improvements in human-preferred generation.

We believe that the transition from feasibility to high-accuracy reward modeling is significant. Only when generative reward models achieve sufficiently competitive discriminative performance can they become viable candidates for widespread adoption in practical reinforcement learning systems.

4. Differences in Research Objectives: Step-Wise Rewards vs. High-Accuracy Final-Image Evaluation

LRM primarily focuses on learning noise-aware reward signals across diffusion timesteps to support Step-wise Latent Preference Optimization (LPO). Its design therefore emphasizes providing usable reward feedback at different intermediate noisy states.

In contrast, DiT-Reward focuses on high-accuracy preference evaluation of completed images and operates on near-clean latents.

Our noise-sensitivity experiments further indicate that reward prediction accuracy decreases as the input noise level increases. We will provide the corresponding ablation results in our supplementary experiments to further justify this design choice.

Therefore, the two works emphasize different objectives: LRM focuses on reward modeling across intermediate diffusion states for step-wise optimization, whereas DiT-Reward focuses on extracting the preference-discriminative capability of generatively pretrained representations to evaluate completed images and guide downstream RL.

Summary

We believe DiT-Reward contributes three mutually reinforcing advances beyond LRM:

  • Architecture and Scaling: Systematically investigating reward representations in modern large-scale MMDiTs and providing evidence of positive reward scaling with stronger generative backbones.
  • Scientific Understanding: Revealing that pretrained generative representations already encode transferable preference information, and characterizing its organization and complementarity across Transformer depths.
  • Practical Value: Advancing diffusion-native reward modeling to competitive accuracy against strong VLM-based reward models, with effectiveness further validated through independent human evaluation.

If LRM established the feasibility of using diffusion models as reward models, DiT-Reward further demonstrates the evaluative potential, scalability, and practical value of modern large-scale generative Transformers.

AuthorsOpenReview ↗

Noise-Level τ Ablation

Thank you for suggesting a noise-level ablation. Following your recommendation, we conducted additional experiments with τ = 0.0, 0.1, 0.5, and 0.8, keeping the backbone, training data, feature layers, and optimization settings unchanged. We include the paper’s original τ = 0.005 setting for reference.

Table. Reward prediction accuracy (%) across noise levels.

τHPDv3 ↑ImageReward ↑HPDv2 ↑PickScore (Pick-a-Pic) ↑
0.077.0666.3986.0865.50
0.005 (original)77.6067.0085.6066.70
0.177.5667.1585.7167.20
0.576.2264.9985.2866.60
0.872.2161.7377.9664.10

Small perturbations can benefit reward prediction, with performance varying only modestly within the low-noise regime. Larger τ reduces the input signal-to-noise ratio and is associated with lower accuracy. Notably, τ = 0.1 modestly improves three of the four benchmarks over our original setting while maintaining comparable HPDv3 performance; we will consider this configuration in future experiments. We thank the reviewer for motivating this informative ablation.

Reviewer_wq7hOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces DiT-Reward, which repurposes pretrained diffusion transformers for text-to-image reward modeling by aggregating features across layers. Its two variants outperform HPSv3 on four preference-prediction benchmarks. When used as the reward in Flow-GRPO, the SD3.5-Large variant yields a policy that human raters prefer to the policy trained with HPSv3.

Critical strengths

The paper demonstrates that pretrained diffusion transformer representations can support effective reward modeling, with improvements over HPSv3 on four preference benchmarks using the same training data mixture. Direct latent scoring also reduces reward inference cost when the policy and reward model share a latent space, making the approach efficient for reward-based optimization of text-to-image models.

Critical weaknesses

The method remains susceptible to reward overoptimization during prolonged RL training, and its prompt-alignment benefits are less consistent than its visual-quality gains. Appendix C reports lower prompt-alignment scores than HPSv3 at later checkpoints under both VLM judges, although human evaluation at step 660 favors DiT-Reward. These results leave uncertainty about whether the method can reliably improve visual quality while preserving prompt fidelity, which is important for practical text-to-image generation.

Questions for authors

Quantifying the agreement between GPT-5, Gemini-3-Flash, and human judgments on the same 2,048 image pairs, particularly for prompt alignment, would help assess the reliability of the automated evaluation.

3 comments
AuthorsOpenReview ↗

Beyond Aggregate Agreement: Human Validation and Evaluation Protocol

Thank you for raising the question of agreement between automated and human evaluations, particularly regarding prompt adherence. We conducted a pair-level analysis of 2,048 image pairs at RL step 660, with five human annotators per pair. As shown in Table 1, human evaluation favors DiT-Reward in overall preference and all four individual dimensions, although the advantage in prompt adherence is smaller than that in visual quality and detail richness. We therefore report overall preference and prompt adherence separately, rather than treating gains in the former as a guarantee of gains in the latter.

Table 1. Human preferences at RL step 660: DiT-Reward vs. HPSv3.

DimensionDiT-Reward winsHPSv3 winsTies
Overall preference1,53844367
Prompt adherence889656503
Visual quality1,497382169
Realism and plausibility1,188535325
Detail richness1,568345135

Each dimension includes 2,048 image pairs, with five annotations per pair. DiT-Reward is preferred when it receives more votes than HPSv3, and vice versa; equal vote counts are treated as ties.

Table 2 reports agreement on pairs where both the aggregated human judgment and the respective automated evaluator express a non-tied preference. GPT–human and Gemini–human agreement on prompt adherence is 60.4% and 63.7%, respectively, lower than their agreement on visual quality and detail richness. Table 2. Agreement between automated and human preferences at RL step 660.

DimensionGPT–human agreementGPT–human pairsGemini–human agreementGemini–human pairs
Overall preference64.4%1,85569.0%1,778
Prompt adherence60.4%98363.7%893
Visual quality68.7%1,16678.9%1,428
Realism and plausibility66.6%1,14169.2%1,329
Detail richness75.5%1,30079.6%1,426

Each comparison includes only pairs where both the aggregated human judgment and the corresponding automated evaluator express a non-tied preference. Eligibility is determined independently for GPT and Gemini. GPT has 2,048 valid pairs; Gemini has 2,044.

To better understand this result, we stratified the pairs by the human preference vote margin, defined as the absolute difference between the vote counts for the two models. Table 3 shows that directional agreement generally increases with stronger human consensus: GPT–human agreement rises from 55.2% at a one-vote margin to 75.4% under unanimous human preference, while Gemini–human agreement rises from 58.1% to 71.6%. Thus, a single aggregate agreement rate obscures differences across levels of human consensus. Furthermore, 68.3% and 73.6% of all GPT–human and Gemini–human disagreements, respectively, involve a tie from one evaluator and a non-tied preference from the other, rather than opposite preferences.

Table 3. Prompt-adherence agreement stratified by human preference vote margin.

Human vote marginGPT–human agreementGemini–human agreement
1 vote55.2% (224/406)58.1% (207/356)
2 votes58.8% (141/240)63.9% (145/227)
3 votes65.7% (109/166)69.7% (115/165)
4 votes66.7% (68/102)69.2% (54/78)
5 votes — unanimous75.4% (52/69)71.6% (48/67)

The vote margin is the absolute difference between DiT-Reward and HPSv3 vote counts among five annotators. Each cell reports agreement percentage (agreeing pairs / eligible pairs), using the same non-tied eligibility rule as Table 2.

Qualitative inspection during training identified approximately 600–700 steps as the best-performing range for generation quality, from which we selected step 660 for detailed evaluation. Both methods were evaluated at step 660 with matched policy-training settings—including optimization steps, global batch size, learning rate, and scheduler—to ensure a comparable evaluation rather than independently selecting each method’s most favorable checkpoint.

Thanks for the author's efforts on addressing my concerns. Though I consider the human-VLM agreement on around 6x% is low, I have no further concern. So, I keep my positive rating.

Thank you for your thoughtful feedback. The limitations of automated evaluation are precisely why we conducted an independent human study on 2,048 image pairs, with five annotators per pair. Aggregating judgments across this sample helps reduce the influence of individual examples and annotation variability, providing human evidence beyond the VLM-based results. We appreciate your careful review and constructive discussion.

Reviewer_dPpzOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

DiT-Reward repurposes pretrained text-to-image diffusion transformers as preference reward models. It adds a small amount of noise to image latents, processes them jointly with text, and predicts a scalar reward using an MLP over concatenated, mean-pooled image-token representations from multiple layers. The VAE and text encoders are frozen, while the DiT backbone and reward head are fine-tuned using the Bradley–Terry loss in the main setting. The paper shows that generative pretraining already provides preference-relevant information and that certain combinations of middle and later layers are useful.

Using the same public preference-data mixture, the SD3.5-Large and Qwen-Image variants reportedly achieve higher accuracy than HPSv3 on all four benchmarks. In a matched Flow-GRPO comparison using an SD3.5-Large policy, the default SD3.5-Large-based DiT-Reward is favored by external VLM evaluations and human evaluations on 2,048 image pairs. Shared-latent reward scoring is also reported to be 1.65× faster than HPSv3 in a batch-size-1 setting.

Critical strengths
  • The paper provides strong evidence connecting improvements in reward modeling to downstream policy performance and human preferences. In Table 1, DiT-Reward-Heavy achieves 79.1% on HPDv3 compared with 76.9% for HPSv3, a 2.2-percentage-point higher point estimate. More importantly, the paper evaluates downstream effects of the default SD3.5-Large-based DiT-Reward using the same SD3.5-Large policy under matched Flow-GRPO settings. At step 660, five ratings are collected for each of 2,048 image pairs. After aggregating the ratings, DiT-Reward wins 1,538 pairs (75.1%, 95% CI: 73.2-76.9%), compared with 443 pairs (21.6%) for HPSv3. The advantage remains under a stricter rule requiring support from at least three raters, with win rates of 73.7% versus 21.2%.

  • The representation analysis provides practical insights into the method's design choices. Training only the reward head on a frozen pretrained SD3.5-Large backbone achieves 72.3% on HPDv3, while full adaptation reaches 77.6%. This comparison helps distinguish the usefulness of pretrained generative features from the additional benefits of preference adaptation. The single-layer and block-pair probing experiments also provide evidence for selecting features from particular network depths.

Critical weaknesses
  • The evidence isolating the contributions of individual design choices is limited. Given that prior work has already explored generative backbones for latent reward modeling, comparisons between single-layer and multi-layer representations under full adaptation, using the same backbone, comparable reward-head capacity, and training budget, would be important for understanding the independent contribution of the proposed design. Similarly, comparing τ=0\tau = 0 with the near-clean noise setting would help establish the benefit of noise injection. The current analysis reveals where preference information is accessible, but does not directly isolate whether the multi-layer design is responsible for the improvements of the final model.

  • The comparisons among SD3.5-Medium, SD3.5-Large, and Qwen-Image are useful as an observed capacity trend. However, they do not isolate the effect of parameter count from differences in pretraining, text encoders, and architectures, particularly in the cross-family comparison. Appendix A does not specify the layer selection for SD3.5-Medium or clearly state whether it uses the same training hyperparameters as the larger variants, limiting the reproducibility and interpretation of the capacity comparison.

Questions for authors
  • Table 1 states that HPSv3 was re-evaluated using its official checkpoint and the official PickScore protocol. Could the authors explain the discrepancy between the resulting 65.1% and the 72.8% reported in Table 4 of the original HPSv3 paper?

  • Was the selection of step 660 for human evaluation predetermined, or was it based on a separate validation procedure?

  • Do the authors have results comparing single-layer versus multi-layer representations, and τ=0\tau = 0 versus τ=0.005\tau = 0.005, using the same fully adapted backbone, comparable reward-head capacity, and the same training budget?

5 comments
AuthorsOpenReview ↗

Tracing the HPSv3 Score: Calibration and Reproducibility

Thank you for pointing out this discrepancy. Pick-a-Pic evaluation involves both human-labeled ties and validation-based threshold calibration, making it more involved than a simple binary preference accuracy calculation. Another user has also reported difficulty reproducing the paper’s Pick-a-Pic score and requested clarification on tie handling and test data in HPSv3 GitHub issue #18. We therefore carefully audited the source data, inference implementation, and evaluation procedure.

According to the PickScore paper and its reference evaluation code, evaluation uses Pick-a-Pic v1’s validation_unique and test_unique splits, each containing 500 image pairs. Each pair includes an original prompt, two images, and a human preference label, including ties. We apply a softmax to the two model scores and predict ties based on their probability difference. The tie-aware metric assigns 1 point for an exact match, 0.5 points when exactly one of the prediction and label is a tie, and 0 points for opposite non-tied preferences. Each model’s threshold is selected independently on validation data and then fixed for testing; test labels are never used for calibration.

The PickScore protocol selects the threshold that maximizes validation accuracy, with the reference implementation using a finite grid search. Since tie-aware accuracy can be sensitive to the threshold, we used a finer 0.05-spacing grid to reduce discretization effects in calibration, at little additional computational cost because the search reuses cached model scores. For a fair comparison, DiT-Reward and HPSv3 use the same data, scoring rule, and threshold grid, with each model’s threshold selected independently on validation data. We have also publicly released all HPSv3 pair-level raw scores and labels, together with evaluation and inference code, in an anonymous Hugging Face repository to enable independent verification.

From these raw scores, HPSv3’s validation-selected threshold is 0.65, yielding 66.0% on validation and 65.1% on test. As a cross-check, applying the reference script’s 0.1-spacing grid to the same scores selects 0.80 and yields 63.5% on test. Thus, the finer grid is empirically more favorable to HPSv3 in this evaluation, increasing its test score by 1.6 percentage points relative to the reference grid. This grid difference does not explain the gap to the 72.8% reported in the HPSv3 paper; the cause of the remaining discrepancy is unresolved, and we provide the released artifacts to support further investigation.

For the RL comparison, qualitative inspection during training identified approximately 600–700 steps as the strongest-performing range for generation quality. Since GPT and Gemini evaluations are less costly than human annotation, we also evaluated checkpoints across the full training curve. Their broadly consistent trends support this observation, indicating that step 660 lies within a broader strong-performing region rather than being an isolated favorable checkpoint. We therefore selected it for detailed human evaluation. Both methods were evaluated at the same step under matched policy-training settings—including optimization steps, global batch size, learning rate, and scheduler—providing a matched-budget comparison.

We also thank the reviewer for the other questions and constructive suggestions. Additional experiments addressing these points are currently underway, and we will share the results and further clarifications as they become available. We welcome continued discussion.

AuthorsOpenReview ↗

Clarifying SD3.5-Medium and Scaling Trends

Thank you for highlighting the missing reproducibility details and the interpretation of the scale comparison. In this discussion, we provide the SD3.5-Medium configuration below and clarify the shared and backbone-specific settings. We will also release the training code and model weights as soon as possible to support independent reproduction.

Currently, the availability of open-source diffusion families with multiple model sizes remains limited. We therefore first compare SD3.5-Medium and Large within the same family. They use the same CLIP-L, CLIP-G, and T5 text-encoder combination, with identical text-encoder and VAE configurations. These components remain frozen during training, reducing some component-level differences. We adopt model-specific optimization settings to accommodate the optimization characteristics of different model sizes, rather than imposing identical hyperparameters across all models.

Qwen-Image differs in text representations and pretraining, but also employs a dual-stream multimodal DiT design. It therefore provides complementary cross-backbone evidence for our central investigation: learning rewards from DiT features. Accordingly, we interpret these results as an observed performance trend across differently sized and pretrained DiT backbones, rather than causal evidence for the isolated effect of parameter count.

SD3.5-Medium training configuration

ItemSetting
Feature blocks[11, 17, 23] (zero-based; outputs of blocks 12, 18, and 24)
Fine-tuningFull backbone fine-tuning; frozen VAE and text encoders; no LoRA
Reward headThree-layer MLP: input features → 1024 → 256 → 1; GELU; dropout 0.1
Learning ratesBackbone: 5e-6; reward head: 1e-4
BatchPer-device batch 8; gradient accumulation 2
OptimizerAdamW; β=(0.9, 0.999); ε=1e-8; weight decay 0; gradient clipping 1.0
ScheduleCosine; warmup ratio 0.05
Noise settingnoise_sigma=0.005; fixed_timestep=None
Training dataHPDv3, ImageReward, and Pick-a-Pic; confidence threshold 0.95
Training durationApproximately 2 epochs
Other settingsBF16; gradient checkpointing; seed 42
AuthorsOpenReview ↗

Noise-Level τ Ablation

Thank you for suggesting a noise-level ablation. Following your recommendation, we conducted additional experiments with τ = 0.0, 0.1, 0.5, and 0.8, keeping the backbone, training data, feature layers, and optimization settings unchanged. We include the paper’s original τ = 0.005 setting for reference.

Table. Reward prediction accuracy (%) across noise levels.

τHPDv3 ↑ImageReward ↑HPDv2 ↑PickScore (Pick-a-Pic) ↑
0.077.0666.3986.0865.50
0.005 (original)77.6067.0085.6066.70
0.177.5667.1585.7167.20
0.576.2264.9985.2866.60
0.872.2161.7377.9664.10

Small perturbations can benefit reward prediction, with performance varying only modestly within the low-noise regime. Larger τ reduces the input signal-to-noise ratio and is associated with lower accuracy. Notably, τ = 0.1 modestly improves three of the four benchmarks over our original setting while maintaining comparable HPDv3 performance; we will consider this configuration in future experiments. We thank the reviewer for motivating this informative ablation.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary and Open-Source Release

We sincerely thank all reviewers and the Area Chair for their time, thoughtful feedback, and constructive discussion.

In our responses, we have provided additional human–VLM agreement analyses, clarified the selection of RL step 660, audited the Pick-a-Pic evaluation protocol, supplemented the noise-level ablation, and detailed the SD3.5-Medium training configuration. We have also distinguished the observed capacity trend from a causal claim about parameter count and discussed its relationship to prior work.

Some questions remain open, particularly matched-backbone comparisons with LRM—including downstream RL effectiveness and inference latency—and single-layer versus multi-layer aggregation under full backbone fine-tuning. The discrepancy with HPSv3’s originally reported Pick-a-Pic score also remains unresolved. We will make every effort to provide further evidence and incorporate these clarifications and limitations into the revised manuscript.

To support independent examination and further research, we have publicly released our SD3.5-Medium, SD3.5-Large, and Qwen-Image reward-model checkpoints, alongside the previously released evaluation artifacts:

We warmly welcome reviewers, the Area Chair, and interested readers to inspect and evaluate these resources. Thank you again for helping us improve the clarity, reproducibility, and scope of this work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.