acceptodds
Under review as a conference paper at ICLR 2027

Diffusion Reward Models

17 views

Abstract

Reward models underpin the alignment of large language models, yet the dominant designs reduce each prompt–response pair to a point estimate or to a distribution from a fixed parametric family. This is at odds with human preference, which is inherently multimodal: the same response can be reasonably judged in many ways, and no single family covers all of them. To better fit this structure, we introduce DRM, a Diffusion Reward Model that recasts reward modeling as conditional density estimation over . Conditioned on a frozen LLM encoder, a lightweight Diffusion Transformer denoises Gaussian noise into a reward vector, placing no parametric assumption on the output distribution and naturally representing its multimodal structure. A single architecture handles both multi-attribute regression and pairwise preference data, and at inference samples form an empirical reward distribution that can be aggregated into a scalar, a variance, or quantiles. Across five benchmarks, DRM matches or surpasses baselines under matched data and backbone, stays competitive with much larger discriminative, distributional, and generative RMs despite its modest training scale, and recovers multimodal reward structure where conventional heads collapse to a point. Uncertainty-aware rejection and lower-confidence-bound (LCB) aggregation further demonstrate that DRM can exploit distributional information beyond a scalar reward to improve reward-model decisions. Downstream RLHF experiments additionally show that using DRM as the training-time reward leads to improved policy performance, directly validating the practical benefit of diffusion-based reward modeling for RLHF training.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Reward Modeling, Reinforcement Learning from Human Feedback

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_WGsqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes DRM, a diffusion reward model that learns conditional reward distributions without a predefined distribution family. It replaces the conventional scalar reward head with a lightweight Diffusion Transformer conditioned on representations from a frozen LLM encoder to capture multimodal reward structure. The same architecture supports multi-attribute regression and pairwise preference learning. At inference, multiple reward samples form an empirical distribution that can be summarized by its mean, variance, or quantiles. Experiments across five reward-model benchmarks show competitive performance, while analyses using repeated human annotations link the learned distributions to human disagreement. Downstream PPO experiments further show improved policy performance.

Critical strengths
  1. Flexible reward modeling. DRM replaces the conventional value head with a diffusion reward head that learns conditional reward distributions without a predefined distributional form, providing flexibility to capture multimodal structures associated with human disagreement.

  2. Compatibility with different supervision formats. A single architecture supports both multi-attribute regression and pairwise preference learning, allowing DRM to use heterogeneous reward data and broadening its applicability across supervision formats.

  3. The proposed method is not only effective in preference modeling but also applicable in downstream LLM RLHF fine-tuning.

Critical weaknesses

A few points require further clarification, as outlined in the questions below.

Questions for authors
  1. The unified architecture supports both multi-attribute scoring and pairwise preference supervision. As a further extension, could the authors discuss the feasibility and potential benefits of joint training with both types of supervision?
  2. Was FsfairX-LLaMA3-RM-v0.1 in Table 2 retrained on the same data ? And, could the authors provide controlled comparisons with QRM-Llama3.1-8B-v2 and URM using the same encoder and training data as DRM, to more directly assess the effectiveness of diffusion modeling ?
  3. All downstream RLHF results in Table 3 rely on GPT-4.1 evaluation. Can the authors provide other evidence that these gains also reflect better alignment with human preferences?
  4. It is unclear whether the MT-Bench gains in Table 3 are robust to sampling randomness. Could the authors specify the sampling settings for evaluation? I also suggest evaluating the same trained policies on IFEval, RACE, GSM8K, or MATH-500 to further assess generalization across tasks.
Reviewer_Ezm1OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This works explores a new reward modeling paradigm beyond conventional discriminative or generative reward models. It proposes Diffusion Reward Model (DRM) which replaces the standard regression or parametric reward heads with a diffusion head. This aims to capture the multimodal and uncertain nature in human preferences. Specifically, DRM trains a conditional diffusion model head on top of a frozen LLM encoder, generating reward distributions that can be aggregated into scalar scores or uncertainty-aware decisions. The framework is also compatible with both pairwise preference and multi-attribute supervision under the same architecture. The experiments cover five reward model benchmarks, downstream RLHF, and additional analysis on utility in cases with human disagreement and uncertainty.

Critical strengths
  1. Analyzes an important and practical problem in preference alignment
  2. The idea of using diffusion head is interesting and novel approach to reward modeling
  3. Broad experiment coverage, including both benchmark accuracies and downstream RLHF performance
Critical weaknesses

I will raise my score if the following can be adequately addressed:

  1. Controlled experiments: Table 2 compares with baselines trained from different backbones, model sizes, datasets, and training scales, which is difficult to attribute performance gains to the diffusion head. Further experiments that control for these differences by using e.g., the same frozen backbone, training dataset, ordering, evaluation configs, etc should be included to isolate the contribution of the proposed pipeline alone
  2. Motivation and method: While the motivation is clear, it’s less intuitive how the proposed method and current results establish it. (a) For pairwise preference training, the diffusion head is trained with a pseudo-reward as supervision, which does not give signals about reward uncertainty or true underlying distribution. (b) Current analysis in Appendix E.2 only compared with relatively weak baselines. Controlled experiments with existing methods, such as QRM and URM, would more directly validate the claimed advantage of diffusion
Questions for authors
  1. How does the choice of pseudo-reward margin affect DRM performance? Perhaps try comparing performance using pseudo margin vs. ground-truth margin— this is available in datasets such as llm-blender/Unified-Feedback
  2. Does the evaluation set in Appendix E.2 (HelpSteer2-Disagreements) have overlap with DRM’s training dataset (HelpSteer2)?
  3. DRM's gains seem to vary across benchmarks, with particularly strong results on RMB Pairwise but weaker on RewardBench-v2 and RM-Bench. What differences between these benchmarks might explain this variation, and why might the diffusion head be more effective in certain settings?
Reviewer_XoqqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes Diffusion Reward Models (DRM), a framework that reformulates reward modeling as conditional distribution estimation rather than deterministic scalar prediction. Motivated by the multimodal nature of human preferences, DRM replaces conventional reward heads with a lightweight Diffusion Transformer conditioned on representations from a frozen LLM encoder. The framework supports both multi-attribute reward regression and pairwise preference learning through denoising and Bradley–Terry objectives. At inference time, multiple diffusion samples approximate the reward distribution, enabling mean-based scoring, uncertainty estimation, and risk-sensitive aggregation.

The authors evaluate DRM on five reward-modeling benchmarks, demonstrating improvements over selected baselines and competitive performance against larger reward models. Additional experiments investigate its ability to capture human disagreement, uncertainty-aware rejection, Best-of-N scaling, and inference efficiency. Downstream PPO-based RLHF experiments further show improvements in policy performance when using DRM as the reward model.

Critical strengths
  1. Novel exploration of diffusion-based reward modeling. The paper explores an interesting and relatively underexplored direction by introducing diffusion models into reward modeling. Instead of producing deterministic scalar rewards, DRM models the conditional reward distribution, offering a flexible way to represent multimodal human preferences and uncertainty. This provides a potentially valuable alternative to conventional reward modeling approaches and opens up interesting directions for future research.

  2. Clear presentation and well-structured empirical investigation. The paper is generally well-written and logically organized, with clear motivations, method descriptions, and experimental settings. The authors provide evaluations across multiple reward-modeling benchmarks, along with analyses of human disagreement, uncertainty-aware decision-making, and downstream RLHF performance. These experiments make the proposed framework relatively easy to understand and assess, while providing preliminary evidence of its practical potential.

Critical weaknesses
  1. Insufficient evidence for the unique benefits of diffusion-based reward modeling and potentially unfair baseline comparisons. DRM relies on an existing reward-model-trained LLM as its frozen encoder, with the main architectural change being the diffusion reward head. However, the comparisons with existing distributional reward models (e.g., QRM and URM) do not clearly control for backbone initialization, training data, and optimization settings. More importantly, the paper lacks comparisons with simpler yet expressive distributional heads, such as conditional Gaussian mixture models. Therefore, it remains unclear whether the reported improvements are attributable to diffusion itself or merely to distributional modeling. This weakens the paper's central methodological contribution, particularly given the additional inference overhead introduced by diffusion sampling.

  2. Limited evidence that the learned reward distributions provide meaningful advantages for downstream RLHF. Although DRM is motivated by capturing multimodal human preferences, the preference-based variant relies on fixed pseudo-reward targets, raising questions about whether its learned distributions genuinely reflect human disagreement. More importantly, the downstream PPO experiments simply average the generated reward samples into a scalar, without exploiting uncertainty or multimodal structure. The improvements from distribution-aware LCB aggregation are also relatively marginal. Consequently, the experiments demonstrate that DRM can produce useful scalar rewards, but do not convincingly establish that modeling the full reward distribution provides substantial practical benefits over conventional reward models.

Questions for authors
  1. Clarification on baseline fairness: The paper claims improvements under matched training data and backbone settings. Could the authors clarify whether QRM, URM, and other distributional baselines were retrained using exactly the same frozen encoder and training data as DRM, or whether their publicly released checkpoints were directly evaluated? If the latter, how can the improvements be specifically attributed to the diffusion reward head?

  2. Necessity of diffusion modeling: Could the authors provide a controlled comparison against a lightweight conditional Gaussian mixture model or another expressive distributional head, using identical backbone representations, training data, and comparable model capacity? Evaluating both reward-ranking performance and distributional fidelity to repeated human annotations would help establish whether diffusion provides benefits beyond conventional distributional modeling.

  3. Practical value of the learned reward distribution: Since the downstream PPO experiments reduce DRM's sampled rewards to their mean, what evidence demonstrates that the learned uncertainty and multimodal structure actually benefit policy optimization? Could the authors compare mean-based rewards against distribution-aware alternatives (e.g., LCB-based rewards) under the same PPO setup? Such results would help determine whether modeling the full distribution offers meaningful advantages beyond improving scalar reward estimation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.