acceptodds
Withdrawn from ICLR 2027

From Soft Targets to Reward Signals: How Assignment and Reward Objectives Interact

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Soft preference targets specify supervision strength, and reward objectives convert that strength into learned reward signals. A central design question remains: how does assigning a fixed set of preference strengths to different response pairs change the rewards produced by different objectives? We introduce assignment geometry to study this interaction. Mean-matched smoothing controls target dispersion, while within-stratum reassignment changes correspondence and preserves the complete target distribution. Across five reward objectives, intact correspondence retains the largest clean preference margins among the compared soft targets within a common accuracy-equivalence budget. Attenuation orderings change with the reward objective, revealing different responses to the same target assignments. Independent reassignments and a related source construction reproduce the retention direction. An attenuation-retention profile compares these combinations through margin magnitude, edit response, and accuracy. Against independently calibrated scaling, APLOT uniform targets deliver additional attenuation on both aggregate and presentation edits. These findings establish a joint design space in which target placement and reward objective shape reward properties beyond preference accuracy.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
reward modeling, preference learning, soft targets, supervision assignment, large language models

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_4XL5OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how soft preference scores are assigned to training pairs and how they interact with the reward model’s training objective. The authors compare uniform, intact, and reassigned soft targets across five reward objectives. Models with similar preference accuracy can still differ in reward margins and sensitivity to edits. Keeping each score with its original pair preserves larger margins across all five objectives, while the best way to reduce edit sensitivity depends on the objective.

Critical strengths

None

Critical weaknesses
  1. The value of comparing uniform, intact, and reassigned targets across five objectives is unclear. Intact targets still retain the largest clean preference margins, a result that prior studies have already suggested.
  2. The paper does not evaluate downstream policy optimization, so it remains unclear whether larger reward margins or lower edit sensitivity lead to better generated responses.
Questions for authors
  1. Intact achieves the highest retention under all five objectives. If practitioners would therefore choose Intact regardless of the objective, could you clarify how the objective-dependent attenuation rankings would affect a practical choice of target assignment?
Reviewer_FXQoOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies how soft preference target assignments interact with reward modeling objectives. The authors introduce assignment geometry, comparing Uniform, Intact, and Reassigned target configurations.

Experiments across five reward objectives (BT, NormBT, BSR, APLOT, and DARM) show that Intact consistently retains larger reward margins than Reassigned, while the effects on edit sensitivity depend on the objective. Additional experiments show that some attenuation effects cannot be explained by simple reward rescaling.

Critical strengths
  1. The paper provides systematic and well-controlled comparisons of soft-target assignments across different reward objectives.

  2. The experiments consistently demonstrate that target correspondence affects learned reward margins, even when the target distribution is preserved.

  3. The calibrated scaling analysis shows that some attenuation effects cannot be explained by simple reward rescaling.

Critical weaknesses
  1. Unclear practical significance of the proposed metrics.

This is my main concern. The paper shows that different target assignments affect reward margins and edit sensitivity, but it is unclear whether these changes are actually desirable for RLHF.

Larger reward margins do not necessarily indicate better reward models, especially on incorrectly ranked pairs. Similarly, reduced edit sensitivity may be beneficial for formatting changes but undesirable when edits affect response quality.

Without establishing how these metrics relate to reward quality or downstream RLHF performance, I find it difficult to assess the practical significance of the findings. I do not think large-scale RLHF experiments are necessarily required, but some evidence connecting these metrics to meaningful learning outcomes would strengthen the paper.

  1. Limited novelty and generality.

The effects of instance-specific soft targets and confidence allocation have been studied previously. The paper extends these ideas to different reward objectives, but the contribution appears relatively incremental.

Moreover, the experiments rely on one backbone and a narrow range of model-generated soft targets, leaving their generality unclear.

Questions for authors
  1. What is the practical interpretation of higher clean-margin retention or stronger edit attenuation? Under what conditions should these changes be considered beneficial for reward modeling or downstream RLHF? Can the authors provide empirical evidence connecting these metrics to meaningful improvements in reward quality or policy performance?

  2. For the edit attenuation experiments, how can we distinguish desirable robustness to irrelevant formatting changes from undesirable insensitivity to genuine changes in response quality? Could the authors evaluate this distinction using edits with validated quality labels?

  3. To what extent do the observed assignment effects generalize to soft targets obtained from independent preference-strength annotations or different model representations?

Reviewer_RTvxOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies reward model learning with soft preference targets. It asks how do learned rewards change as the preference strength/confidence for different response pairs change. The experiments generate a preference strength for each pair, and study how the learned rewards change when different preference strengths are used: 1- intact (actual preference strength), 2- reassigned (shuffled strengths) , or 3- uniform (assigning equal preference strengths across all data). The results show the choice of preference strengths affects the learned rewards, and largest preference margin (i.e., difference between positive and negative sample rewards) is obtained under intact preference strengths.

Critical strengths

The reviewer could not spot significant strengths that supports acceptance in ICLR.

Non-critical strengths include careful controls and detailed report of experimental results.

Critical weaknesses

The main finding is quite expected, and utility of the results and the study is not clear. If the confidence carries useful information about each pair, shuffling them turns them into noise. And a model trained on such noise would have less confident reward model (i.e., would have smaller margin d). This is expected and the reviewer wonders what is the significance/surprise in the experiments confirming this expected behaviour. It is not even clear if larger or smaller d is good, when the date itself has bounded confidence (and the success measure is then not quite clear).

Questions for authors

Why do experiments generate the confidence data based on the features of a pertained model, rather than using a frontier AI model to generate confidence values?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.