- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes CARE, a method for generating adaptive rubric-based rewards for reinforcement learning on open-ended tasks. It first combines the current policy’s responses into a candidate answer, then creates a near-miss version with a few deliberate errors or omissions. Comparing these answers helps identify specific criteria that guide the final rubric, which is used to score the original responses for GRPO training. A frozen copy of the initial model handles all feedback stages, without human annotations, external reference answers, or a stronger teacher during training. Experiments show improvements in both response ranking and downstream performance on medical and research question answering. The main contribution is using near-miss comparisons to make rubrics more sensitive to meaningful differences in response quality.
Critical strengths
-
CARE uses a near-miss answer with targeted errors or omissions to identify specific requirements that generic rubrics may overlook. Keeping the two answers similar in style and length helps focus criterion generation on substantive differences.
-
Evaluation connects rubric quality to training outcomes. The paper tests both preference-ranking accuracy and downstream improvements after GRPO training. Its Positive-only ablation shares the same training and scoring pipeline but removes the near-miss and comparison stages, providing a useful comparison for assessing the contribution of contrastive anchoring.
Critical weaknesses
- Limited validation of the synthetic contrasts. The same frozen model generates both the pseudo-reference and its near-miss, without independently checking their quality. Although the paper reports better reward discrimination, it does not directly establish whether the injected errors reflect common, important failures in actual policy rollouts. The benchmark gains are encouraging, but the proposed mechanism remains insufficiently validated.
- Reliance on a single evaluation judge. All downstream policy evaluations use GPT-5.6 Luna, including both benchmark scoring and pairwise comparisons. Agreement between these evaluation formats therefore does not establish robustness across judges. Independent human assessment or additional judges would strengthen the evidence for genuine quality improvements.
- CARE adds near-miss generation and criterion extraction to the Positive-only pipeline, requiring extra inference at every rollout group. Without a compute-matched baseline, it is difficult to determine how much of the improvement comes from contrastive anchoring rather than the additional feedback-generation budget.
Questions for authors
- How do you verify that the generated near-misses are genuinely worse than the pseudo-references and reflect important errors in actual policy rollouts?
- Have you compared CARE with a compute-matched Positive-only baseline to separate the benefit of contrastive anchoring from that of additional inference?
Thank you for your careful comments
Since you have claimed some of the issues, i decide to increase my score. I find the additional experimental results convincing.