- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies how soft preference scores are assigned to training pairs and how they interact with the reward model’s training objective. The authors compare uniform, intact, and reassigned soft targets across five reward objectives. Models with similar preference accuracy can still differ in reward margins and sensitivity to edits. Keeping each score with its original pair preserves larger margins across all five objectives, while the best way to reduce edit sensitivity depends on the objective.
Critical strengths
None
Critical weaknesses
- The value of comparing uniform, intact, and reassigned targets across five objectives is unclear. Intact targets still retain the largest clean preference margins, a result that prior studies have already suggested.
- The paper does not evaluate downstream policy optimization, so it remains unclear whether larger reward margins or lower edit sensitivity lead to better generated responses.
Questions for authors
- Intact achieves the highest retention under all five objectives. If practitioners would therefore choose Intact regardless of the objective, could you clarify how the objective-dependent attenuation rankings would affect a practical choice of target assignment?