- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies reliable evaluation of rewritten erroneous LLM responses. The authors introduce RHContrast-Bench, leveraging 1151 long-form answer lineages with matched controls for auditing reward hacking risks. The benchmark supports traceable local-defect constructions with joint analysis of reference and defective answers, connecting 3 complementary comparisons. The method is evaluated across 7 LLM judge families, scalar and pairwise reward models, grounding verifiers, and human-reviewed reference judgements.
Critical strengths
- It identifies a concrete weakness in current evaluation methods that could potentially lead to reward hacking.
- A concrete example of restaurant overview is provided to help understand the motivation easily.
- The writing is clear and easy to follow.
Critical weaknesses
Overall I appreciate the motivation behind this work and find the methodology interesting, but I feel the contributions and empirical evidence are not yet sufficiently compelling. I think it would be interesting to connect the study with recursive self-improvements in LLMs. Iterative response refinements could be one direction to further extend the experiments.
Questions for authors
Overall I appreciate the motivation behind this work and find the methodology interesting. The design is rational, but the contributions and empirical evidence are not yet sufficiently compelling. I believe this paper would benefit from further development and stronger empirical evidence. Especially, I think it would be interesting to connect the study with recursive self-improvements in LLMs. Iterative response refinements could be one direction to further extend the experiments.
4 comments
Response to Reviewer 19Nz (2)
Furthermore, the same held-out evaluation showed that an individual judge could not only assign a relatively more favorable score change to the defect branch but also increase the score of the defect answer itself. We use ρ to denote the score change of the treated defect answer relative to its original version, and κ to denote this change minus the corresponding change in the reference branch. Grok’s results are shown below; changes for each question were first averaged over presentation orders and then averaged with equal weights across the scoring rules within each track:
| Grok-4.3 | Comparable questions | Absolute gain for defect answers ρ | Relative interaction κ | Questions with increased/unchanged/decreased scores |
|---|---|---|---|---|
| N track | 39 | +0.2651 | +0.5406 | 15/20/4 |
| E track | 29 | +0.6013 | +0.7710 | 19/7/3 |
Grok showed positive ρ and κ in both tracks: defect answers received higher scores than their own original versions, and their score changes were more favorable than those of the matched reference branch. In the E track in particular, defect answers on 19/29 questions actually gained points after verification endorsement was added. This provides direct sample-level observations that ‘answers retaining defects can still obtain absolute scoring gains,’ complementing cases in which positive κ arises solely from a smaller relative penalty. This phenomenon is model-dependent: the mean score of the E-track defect branch still decreased for the other six judges. Grok has not yet covered all 50 questions, so these results are reported as descriptive findings from the completed pairs.
In addition, after excluding the 34 identified items of semantic concern, 586 questions remained in the original judge evaluation. κ_N changed from −0.1534 to −0.1421, with a 95% confidence interval of [−0.1699, −0.1145], and all seven judges retained negative results. This supports the robustness of the main original N-track finding to the identified semantic issues.
Together, these results strengthen the contribution of this work: matched controls can distinguish among responses such as ‘the defect answer gains points, but the reference answer gains more,’ ‘the defect answer receives a smaller relative penalty,’ and ‘the defect answer gains points in absolute terms and obtains a relative advantage.’ The native-RM and judge held-out experiments provide validation across settings, while Best-of-N provides evidence related to actual candidate selection. The direction of the N-track result in the new held-out sample differs from that in the original evaluation. We treat this as an additional finding on model and sample dependence and report the results separately for each model; the positive E-track interactions for multiple judges and Skywork support the relative scoring pattern reported in the original paper.
Response to Reviewer 19Nz (3)
Comment 2: Especially, I think it would be interesting to connect the study with recursive self-improvements in LLMs. Iterative response refinements could be one direction to further extend the experiments.
Response: This suggestion identifies an important application of the matched-control framework. When answers are repeatedly rewritten based on reward feedback, an increase in reward may result from substantive improvements in content, changes in expression, or verification endorsement. Our framework can compare the reference and defect branches at each iteration, check whether the target defect remains, and simultaneously record absolute reward changes, relative scoring advantages, and comparisons with the original reference answer, thereby distinguishing genuine quality improvements from reward gains that preserve the defect.
The present Best-of-N experiment extends validation from scoring fixed texts to reward-based selection at inference time, supporting a difference in relative selection risk between the two RMs under the tested setting. It evaluates selection from a fixed candidate pool and does not yet include a feedback-driven loop of iterative rewriting. We will clarify this connection in the discussion, present iterative rewriting as an extension of the framework, and distinguish the selection behavior validated so far from the process of recursive self-improvement.
Follow-up on Rebuttal
Dear Reviewer,
Thank you for your time and effort in reviewing our paper.
We have provided detailed responses to your concerns, including additional experimental evidence.
As the discussion period is approaching its end, we would greatly appreciate it if you could take a moment to review our responses.
Thank you again for your time and consideration.
Best regards, The Authors
Response to Reviewer 19Nz (1)
Comment 1: Overall I appreciate the motivation behind this work and find the methodology interesting. The design is rational, but the contributions and empirical evidence are not yet sufficiently compelling. I believe this paper would benefit from further development and stronger empirical evidence.
Response: We thank the reviewer for recognizing the motivation and design of our study. The core contribution is to use matched controls comprising reference and defect answers from the same source to distinguish among ‘a score increase for the defect answer,’ ‘a relative advantage for the defect answer,’ and ‘the defect answer surpassing the original reference answer.’ For example, in the original N track, the reference branch gained +0.3591 points on average and the defect branch gained +0.2056 points, yielding κ_N = −0.1534 when the reference-branch gain is subtracted from the defect-branch gain. Looking only at the score increase for the defect answer would miss the fact that the reference branch benefited more. Our framework can therefore test whether reward increases reflect a shared response to quality or an additional relative advantage for the defect answer.
To strengthen the experimental evidence, we added evaluations of actual candidate selection, native reward scores, and independent held-out samples, connecting differences in controlled scoring to actual selection behavior:
The held-out sample was frozen before evaluation, with overlaps and near-duplicates of previously evaluated items excluded. The two RMs completed all 1200 native-score evaluations; the seven judges returned 2022/2100 valid responses in total. The table above lists the three judges with complete E-track coverage of all 50 questions and confidence intervals above 0. Skywork’s positive κ_E took the form of a smaller relative penalty for the defect branch.