- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper focuses on a research question: whether a reward model that performs well on static evaluationcan measure policy improvement reliably during RL training. To answer this question, the authors train Qwen3-8B with GRPO to maximize RoleRM scores and compares reward scores with evaluations at different checkpoints. The findings are that RoleRM scores increase to nearly the maximum, while character authenticity declines according to GPT-4o and human evaluations.
Critical strengths
The motivation is clear and I think the research question is also worth studying. A reward model that perform well on static comparisons may not reliably measure policy improvement when the policy is trained to maximize its scores. (Although I think experiments are not convincing, see 'Weakness' below) the experiment results could provide some interesting evidence of such phenomenon: RoleRM scores increase during training,but GPT-4o and human annotators give lower ratings for character authentiicity.
Critical weaknesses
I am not convinced by the depth of the contribution and some parts of the experiments for the reasons as follows.
First, I think the paper needs to make its contribution more convincing (beyond the observed failure). The reward overoptimization, the static reward-model accuracy and downstream performance gap have already been studied. It is not clear to me what new insights the role-playing results provide. Reward saturation, identical group rewards, and increasingly generic responses are some interesting observations, but the authors do not clearly explain how these observations relate to the decline of character authenticity. I would like to see a more focused analysis of the observations in this paper. So far I think this work is incomplete.
Second, I also have some concerns about the experiment comparisons. For example, the main experiments use batch size 64 and 500 training steps, the random-reward and stronger KL experiment use batch size 8 and 150 steps. The longer completion experiments also use a different training subset. I am not sure whether these results still hold under the main setting. it would be better to keep the training settings consistent, and I would like to see additional experiments to support the conclusions.
Questions for authors
See the 'Weaknesses' part.
-
Can the authors provide more analysis of why character authenticity decline? I think the current observation are not enough to show what is new beyond existing studies on reward overoptimization.
-
Could the authors repeat the control experiments with the same batch size, training steps, and data selection as the main experiments?