- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes DRM, a diffusion reward model that learns conditional reward distributions without a predefined distribution family. It replaces the conventional scalar reward head with a lightweight Diffusion Transformer conditioned on representations from a frozen LLM encoder to capture multimodal reward structure. The same architecture supports multi-attribute regression and pairwise preference learning. At inference, multiple reward samples form an empirical distribution that can be summarized by its mean, variance, or quantiles. Experiments across five reward-model benchmarks show competitive performance, while analyses using repeated human annotations link the learned distributions to human disagreement. Downstream PPO experiments further show improved policy performance.
Critical strengths
-
Flexible reward modeling. DRM replaces the conventional value head with a diffusion reward head that learns conditional reward distributions without a predefined distributional form, providing flexibility to capture multimodal structures associated with human disagreement.
-
Compatibility with different supervision formats. A single architecture supports both multi-attribute regression and pairwise preference learning, allowing DRM to use heterogeneous reward data and broadening its applicability across supervision formats.
-
The proposed method is not only effective in preference modeling but also applicable in downstream LLM RLHF fine-tuning.
Critical weaknesses
A few points require further clarification, as outlined in the questions below.
Questions for authors
- The unified architecture supports both multi-attribute scoring and pairwise preference supervision. As a further extension, could the authors discuss the feasibility and potential benefits of joint training with both types of supervision?
- Was FsfairX-LLaMA3-RM-v0.1 in Table 2 retrained on the same data ? And, could the authors provide controlled comparisons with QRM-Llama3.1-8B-v2 and URM using the same encoder and training data as DRM, to more directly assess the effectiveness of diffusion modeling ?
- All downstream RLHF results in Table 3 rely on GPT-4.1 evaluation. Can the authors provide other evidence that these gains also reflect better alignment with human preferences?
- It is unclear whether the MT-Bench gains in Table 3 are robust to sampling randomness. Could the authors specify the sampling settings for evaluation? I also suggest evaluating the same trained policies on IFEval, RACE, GSM8K, or MATH-500 to further assess generalization across tasks.