acceptodds
Under review as a conference paper at ICLR 2027

A Preference Is a Spectrum, Not a Scalar: PRISM for Structured Ordinal Reward Modeling

Abstract

Pairwise preferences record which response wins, but discard information an annotator often knows: which aspects separate the responses, how large the gap is, and why the rejected response fails. We study whether this discarded structure, rather than model capacity, limits reward modeling. We re-audit the Skywork-80K preference prompts into five aspect ratings, an ordinal preference strength, and a 17-way error taxonomy, yielding 66,605 valid pairs. We then train PRISM (Preference Refraction Into a Structured Mixture), a lightweight head on a frozen public reward model. PRISM uses sample-conditioned reward atoms, per-aspect ordinal distributions, and a coupled pairwise likelihood with explicit tie mass. A separate prompt-gated verifier handles programmatically checkable constraints without changing rankings when the gate is inactive. On RewardBench 2, the trained head alone reaches 0.855 with Llama-3.1-8B and 0.814 with Qwen3-8B, exceeding same-backbone published results of 0.841 and 0.809. The full system reaches 0.872 and 0.827, respectively. The gains persist from 0.6B to 8B, with a leak-free 5-fold result of 0.8707 ± 0.0011 on Llama. The verifier covers 23/25 IFEval constraint families and transfers to RewardBench-v1. These results show that richer supervision can improve reward models without enlarging the backbone.

Primary area:
foundation or frontier models, including LLMs
Keywords:
reward modeling, preference learning, structured supervision, ordinal regression, multi-aspect evaluation, RLHF, instruction following, verifiable constraints

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_KqqbOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

Pairwise preferences discard important information such as the aspects that separate the response, the size of the gap and limitations of the rejected response. They re-audit Skywork-80K, an existing preference dataset. Specifically, it uses a per-aspect ordinal distribution, as well as prompt-gated verifiers. Using these methods, it shows improvements on Llama and Qwen based models on RewardBench 2.

Critical strengths
  1. The improvements over Bradley-Terry baseline models are consistent and good.
Critical weaknesses
  1. The approach of the method is very similar to ArmoRM “Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts” (EMNLP, 2024), which the authors cite but do not clearly explain the difference. Specifically, ArmoRM trains a prompt-gated head to determine how various HelpSteer attributes with 5 levels of distribution (NAACL 2024) are considered, which is exactly what’s considered in Figure 3. Even the concept of preference strengths (1/2/3) in Table 1 is directly taken from HelpSteer2-Preference “HelpSteer2-Preference: Complementing Ratings with Preferences” (ICLR, 2025) without acknowledgement or differentiation. Similarly, the concept behind “why rejected loses” in Figure 3 was also done in HelpSteer2-Preference paper. The poor positioning of this work among prior works makes it challenging to understand the genuine contributions of the paper.
  2. The paper claims to use 5 other benchmarks alongside RewardBench 2 (see line 148) but only primarily reports results on RewardBench 2 (there was a small mention at line 375 but it does not show how the scores compare to baseline models in a table that is typically expected) It also claims to “summarize(s) the technical differences among these paradigms (of evaluation benchmarks)” in Appendix A but did not do that.
Questions for authors
  1. (Not a question but some general advice for authors) Overall, the paper reads to me like it was either primarily generated by an LLM (way too many em-dashs and weird use of italics) and/or it was written by authors who have not had much experience writing ML papers. Both are fine (e.g. LLM use is disclosed and all authors had to start from somewhere) but I think the current paper is not written well enough to be accepted (see weaknesses, especially weakness 1).
Reviewer_3hYzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper primarily investigates the issue that in preference responses, the judgment of which option is better relies only on a single score without giving the reason for choosing it, and aims to study more fine-grained reasons. Therefore, it extends the preference data with evaluation criteria for making choices, including five absolute dimensions: helpfulness, correctness, coherence, complexity, and verbosity; as well as the degree of winning {1, 2, 3} (1 = close tie, 3 = decisive); and 17 categories of error labels that record why the rejected answer failed. Specifically, a lightweight MLP classification head (350 million parameters) is trained and attached to a frozen Skywork-Reward-V2 reward backbone. To better classify according to more fine-grained levels, 8 cross-attention heads are set up in the MLP, with each attention head focusing on different aspects, thereby satisfying more detailed labels. In terms of experimental setup, the backbone model adopts the frozen Skywork-Reward-V2, based respectively on Llama-3.1-8B and Qwen3-8B, while also validating Qwen3 backbones with different parameter sizes of 0.6B, 1.7B, and 4B. The training data comes from secondary re-annotation of the public dataset Skywork-80K, yielding 66,605 structured preference samples, split into 20,000 training samples and 2,000 validation samples. The main test set is RewardBench 2, with a total of 1,865 samples, divided into 6 equally weighted evaluation domains: 1. Factuality; 2. Precise-IF precise instruction following (examining compliance with hard, verifiable constraints such as format and quantity); 3. Math; 4. Safety; 5. Focus topic relevance; 6. Ties tie sample ranking. The experimental results show that the performance improvement is highly concentrated in the Precise-IF precise instruction following domain, while the metrics for factuality, math, safety, and topic relevance show almost no change. Across all sizes of Qwen3-0.6B, 1.7B, 4B, and 8B, the entire PRISM approach outperforms the corresponding original Skywork-V2, with an improvement of +4.5 to +5.5 points, and the gains are more pronounced for smaller models.

Critical strengths

The advantage of this paper lies in its adoption of a lightweight approach: only a lightweight head of about 350 million parameters is attached to a frozen public reward backbone, and the source of the gain is that the supervision information becomes richer, rather than the backbone becoming larger. Although the symbolic verifier behind it is merely rule matching, it is gated by prompts and only takes effect on prompts containing checkable commitments, while keeping the ranking unchanged in domains without checkable constraints. Also commendable is its use of structured multi-head attention to approximate more fine-grained preference dimensions: the reward dictionary allows the heads to focus on which evidence in the response is important, the five dimension-wise ordered heads allow different quality dimensions to each output an ordered distribution, and the prompt-conditioned router determines which dimension is more critical for the current question, rather than compressing all dimensions into a single score as scalar BT does.

Critical weaknesses

The paper reports cross-dataset transfer results in Section 5.4 and Appendix F: the validation signal transfers to RewardBench-v1 (0.938), IFEval covers 23/25 constraint families, and the symbolic verifier achieves 0.70 under 4-fold family-CV. However, these numbers are all reported in isolation, without any baseline comparison under the same conditions, and RewardBench-v1 is an old benchmark that has already been evaluated by a large number of works, with many publicly comparable numbers available.

Questions for authors

The paper reports in Appendix I that the from-scratch trained PRISM head achieves only 0.769 on RewardBench 2 (0.786 after adding the verifier), which is even worse than the base, with Precise-IF at only 0.425 → 0.531, whereas the headline system (the version distilled from Skywork-V2) reaches 0.855 / 0.872. The paper's explanation for this is that "the headline system is the distilled trained reward model." This leads to a problem: do PRISM's main gains actually come from the head, or from the preference representations already learned by the Skywork-V2 backbone? Specifically, if the structured supervision itself is the source of the gain, then the same head, when attached to a backbone that has not undergone preference training (i.e., trained from scratch), should also outperform the corresponding base. However, Appendix I shows that the from-scratch version achieves only 0.769, which is even lower than the frozen Skywork-V2 base's 0.792; Precise-IF shows almost no improvement when trained from scratch (0.425 → 0.531), whereas the headline version improves dramatically (0.600 → 0.800). This suggests that the headline's gains may mainly come from the weights inherited through distillation, rather than from the structured supervision.

Reviewer_UCuZOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces PRISM, a structured reward-modeling framework that combines multi-aspect ordinal distributions, pairwise coupling, and error-tag supervision on frozen reward-model backbones, supplemented by a prompt-gated symbolic verifier.

Critical strengths
  1. The paper explores a meaningful question: whether richer supervision about quality dimensions, preference strength, and failure modes can improve reward modeling beyond binary preference labels.

  2. The separation between learned reward scoring and prompt-gated symbolic verification is practical, with ranking preservation guaranteed when the verification gate is inactive.

  3. The evaluation includes multiple backbone sizes, repeated cross-validation, matched supervision controls, and held-out constraint-family analyses that help identify the method’s empirical scope.

Critical weaknesses
  1. The connection between the described training recipe and the headline results is unclear. Appendix I identifies the headline system as a distilled model, while a fresh Qwen head trained on the re-annotated data achieves only 0.769/0.786 without/with verification, versus the headline 0.814/0.827; the teacher, distillation targets, and training procedure need to be specified to establish reproducibility and attribution.
  2. The matched controls do not establish the benefit of the proposed structured architecture over scalar reward modeling. In Table 6, scalar BT and ordinal coupling perform similarly with identical supervision (0.7736 versus 0.7744), while removing error-tag supervision improves performance to 0.7850, weakening the justification for two central components.
  3. The evidence supports a narrower improvement than the paper’s broad reward-modeling motivation suggests. Almost all gains occur in Precise-IF, while Ties accuracy remains unchanged or slightly decreases relative to the local base; moreover, mapping the verifier to 23/25 IFEval constraint families does not substitute for measuring its accuracy on those constraints.
Questions for authors

Please refer to the weakness

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.