acceptodds
Under review as a conference paper at ICLR 2027

REWARDING THE WRONG CHARACTER: EVALUATOR VALIDITY UNDER POLICY OPTIMIZATION

Abstract

Reward models are used both to evaluate outputs and to optimize the policies that generate them. This dual role creates a validation problem: strong static discrimination may coexist with misleading assessments of policy progress. We investigate this problem in role-playing, where character authenticity can be assessed independently by human annotators. RoleRM achieves 95.3% accuracy on static character contrasts, yet optimizing Qwen3-8B against its fixed scoring rule increases held-out reward from 13.62 to 17.72 out of 18 while human authenticity falls from 3.54 to 1.47 out of 5. On 50 matched prompts, RoleRM favors the final response on 43, whereas humans favor the initial response on 47. We term this pattern reward–fidelity inversion. Reward saturation, loss of within-group score variation, and response homogenization accompany the divergence; stronger KL regularization attenuates it, and a second policy–reward configuration exhibits the same direction of change. We organize these measurements into a trajectory-based evaluator validation protocol that pairs fixed prompts and checkpoint comparisons with independent outcome judgments. The central lesson is that holding out prompts does not make the reward used for optimization an independent measure of progress.

Primary area:
reinforcement learning
Keywords:
Reward Modeling, Reinforcement Learning, Reward Hacking, Role-Playing Language Models, Model Evaluation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_GSDAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper focuses on a research question: whether a reward model that performs well on static evaluationcan measure policy improvement reliably during RL training. To answer this question, the authors train Qwen3-8B with GRPO to maximize RoleRM scores and compares reward scores with evaluations at different checkpoints. The findings are that RoleRM scores increase to nearly the maximum, while character authenticity declines according to GPT-4o and human evaluations.

Critical strengths

The motivation is clear and I think the research question is also worth studying. A reward model that perform well on static comparisons may not reliably measure policy improvement when the policy is trained to maximize its scores. (Although I think experiments are not convincing, see 'Weakness' below) the experiment results could provide some interesting evidence of such phenomenon: RoleRM scores increase during training,but GPT-4o and human annotators give lower ratings for character authentiicity.

Critical weaknesses

I am not convinced by the depth of the contribution and some parts of the experiments for the reasons as follows.

First, I think the paper needs to make its contribution more convincing (beyond the observed failure). The reward overoptimization, the static reward-model accuracy and downstream performance gap have already been studied. It is not clear to me what new insights the role-playing results provide. Reward saturation, identical group rewards, and increasingly generic responses are some interesting observations, but the authors do not clearly explain how these observations relate to the decline of character authenticity. I would like to see a more focused analysis of the observations in this paper. So far I think this work is incomplete.

Second, I also have some concerns about the experiment comparisons. For example, the main experiments use batch size 64 and 500 training steps, the random-reward and stronger KL experiment use batch size 8 and 150 steps. The longer completion experiments also use a different training subset. I am not sure whether these results still hold under the main setting. it would be better to keep the training settings consistent, and I would like to see additional experiments to support the conclusions.

Questions for authors

See the 'Weaknesses' part.

  1. Can the authors provide more analysis of why character authenticity decline? I think the current observation are not enough to show what is new beyond existing studies on reward overoptimization.

  2. Could the authors repeat the control experiments with the same batch size, training steps, and data selection as the main experiments?

Reviewer_Q1i5OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies whether a role playing RM remains a reliable evaluator when policies are trained to maximize its scores. RoleRM achieves 95.3% accuracy on constructed reference comparisons, yet optimization increases reward while human rated character authenticity decreases across three Qwen runs, significantly in two. A second policy reward configuration shows a similar discrepancy.

Critical strengths
  • The central discrepancy issue argued by the authors is backed by human evidence. In two additional Qwen runs, mean human authenticity falls from 3.16 to 1.73 and 1.99 out of 5, with both declines significant after multi-comparison correction, while RoleRM fidelity rewards remain near their ceiling.
    • The separate llama experiment also shows increasing reward along with a substantial human rated decline. I find this replication valuable as it shows that the discrepancy extends beyond a single training run or policy reward config.
  • Authors' interpretation of the results is carefully qualified. They acknowledge that degradation varies across seeds, that the main Qwen run's human-rated decline is not statistically significant, and that automated judgments depend on the evaluation protocol. They also disclose config differences that limit conclusions from the controls.
Critical weaknesses
  • Scoring interface effects remain unclear. Training limits instructions and responses to 200 and 400 chars which seems way too short for today's standards. And scoring output budgets also differ across some evaluations.
  • I don't think we can claim that a successful parsing establish these restrictions preserve the relevant character information. Without matched full text scoring and clearer alignment with what human annotators saw, the practical scope of the finding remains unclear to me.
  • The experiments don't seem to explain the causes of failures clearly. In the main static test, mismatched responses differ in both character identity and relevance to the question, so high accuracy may partly reflect detecting irrelevant answers. The tests that preserve content while changing character voice address this concern, but contain only 20 pairs each. Also, control runs use different batch sizes and training durations from the main runs, and their human evaluation excludes the untrained base policy. These differences make it difficult to determine which training choices drive the decline in authenticity or prevent it.
Questions for authors
  • Please clarify, exactly which text did each evaluator and the human annotators receive? Does reward authenticity disagreement persist when identical complete responses are scored using a common, sufficiently large scoring budget?
  • How were the 50 prompts and 20 characters selected, and were these characters present in training? Do the conclusions from the final human evals remain statistically significant when uncertainty is estimated across characters rather than individual prompts?
  • If the random reward and stronger KL controls use the same batch size and training duration as the main run, how does their authenticity compare with the base policy when all are rated together in the same human evaluation? I think this experiment must be done to claim the main conclusion fairly.
Reviewer_WfwJOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper examines whether static reward discrimination predicts progress when role-playing policies are optimized against that reward. It follows Qwen3-8B under GRPO using fixed held-out prompts, automated reward and GPT-4o scores, and blinded human authenticity judgments. Three matched 500-step Qwen runs attain high reward while receiving lower human endpoint means than a shared base policy; two declines are significant after correction, whereas the main run's smaller decline is not. Additional experiments examine reward compression, regularization, reward subsets, judging protocols, and a Llama configuration. The results support the narrow conclusion that rising optimized reward does not guarantee improved human-rated authenticity in this pipeline.

Critical strengths
  1. The central observation has meaningful human support. The unified Qwen comparison uses a common base response set and the same 50 prompts, with three blinded ratings per response. Human means decrease by 0.26, 1.43, and 1.17 points across the three matched training runs, with the latter two significant after Holm correction. The nonsignificant main-run result indicates heterogeneous effect sizes; it does not invalidate the demonstrated counterexamples to reward growth guaranteeing authenticity improvement. [pp. 3–5, 10]

  2. The paper carefully separates several forms of evidence. Static discrimination, fixed-reference comparisons, reward concentration, and independent human outcomes are not treated as interchangeable. The controls include constant/random reward, stronger regularization, reward subsets, and a second policy–reward configuration. Reporting the weaker, order-sensitive pairwise GPT-4o results is particularly useful, and the authors appropriately avoid claiming that reward compression alone causally explains the human decline. [pp. 5–7, 14–18]

Critical weaknesses
  1. The evaluated reward interface is not yet precisely identifiable. The paper attributes RoleRM to Ding et al.'s CIP model, whereas its six-dimensional generative scoring interface appears to match the RoleRM released with Crab. The exact checkpoint/revision identifiers, loading classes, and the meaning of RoleRM-8b-CF are unspecified. This may be a correctable citation or naming issue, rather than an experimental error, but it matters for reproducing the pipeline and interpreting the second configuration. [pp. 1–3, 8–11]

  2. The measurement boundary needs a more explicit account. Reward instructions/responses are truncated to 200/400 characters, but matching truncation for humans is unspecified; human authenticity emphasizes recognizable voice, whereas the reward fidelity sum also includes Role Knowledge. The paper already separates observed disagreement from causal attribution, which is appropriate. Clarifying exactly what each instrument sees and measures would help readers distinguish pipeline-level disagreement from disagreement on a common response span and construct. This qualification does not negate the replicated human evidence or require a new causal mechanism. [pp. 7, 10–11]

Questions for authors
  1. Please identify the exact repositories/revisions and loading classes for RoleRM-8B and RoleRM-8b-CF, reconcile their relationship to the cited CIP model and Crab's generative evaluator, and provide the complete scoring template/parser. Which scoring-output budget was used for each directly compared reference and policy response set? [pp. 1–3, 8–11, 14]

  2. What response spans and profile information did humans and each automated interface receive? How do the human voice-authenticity rubric and the reward's Role Language/Role Knowledge dimensions align? Please state which comparisons use identical material and which should be interpreted only as pipeline-level outcome disagreement. Any existing matched-input analysis would help resolve this boundary. [pp. 10–11]

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.