acceptodds
Under review as a conference paper at ICLR 2027

Rubrics Learn from Near-Miss: Contrastive Anchoring for Rubric Evolution

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) has driven strong post-training gains on tasks with verifiable answers, such as maths and code generation. Yet, extending the method to open-ended generation remains challenging, as ground-truth labels are often unavailable and human annotation is costly for these tasks. Rubric-based methods offer a promising alternative by decomposing response quality into explicit criteria. Recent approaches adapt these criteria during training, but adapted criteria often describe properties that most sampled responses already share, rather than what separates stronger responses from weaker ones. A central challenge is therefore to generate rewards that discriminate among the current policy's responses near this decision boundary, without labelled feedback. We propose Contrastive Anchoring for Rubric Evolution (), a self-improvement method for non-verifiable tasks that requires no human annotations, reference answers, or a stronger teacher model. derives criteria by contrasting a synthetic candidate response against a near-miss variant, surfacing properties that separate stronger from weaker responses on-policy at each point in training. outputs are preferred 68–73% of the time compared to the base model, with consistent gains across HealthBench, ResearchQA, and three models (Gemma-3-4B-it, Qwen3-4B, Qwen3-8B). On both benchmarks, CARE improves over the base model by 19–22%, more than doubling the gains of the strongest rubric baseline. We further show that the gains from on ResearchQA remain after excluding citation coverage, indicating improvements in other aspects of response quality. Overall, CARE shows how near-miss contrastive anchors can provide discriminative feedback to evolve rubrics on-policy for self-improvement.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Open-ended Evaluation, Self-improvement, Rubric Generation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_S5dDOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes CARE, a method for generating adaptive rubric-based rewards for reinforcement learning on open-ended tasks. It first combines the current policy’s responses into a candidate answer, then creates a near-miss version with a few deliberate errors or omissions. Comparing these answers helps identify specific criteria that guide the final rubric, which is used to score the original responses for GRPO training. A frozen copy of the initial model handles all feedback stages, without human annotations, external reference answers, or a stronger teacher during training. Experiments show improvements in both response ranking and downstream performance on medical and research question answering. The main contribution is using near-miss comparisons to make rubrics more sensitive to meaningful differences in response quality.

Critical strengths
  • CARE uses a near-miss answer with targeted errors or omissions to identify specific requirements that generic rubrics may overlook. Keeping the two answers similar in style and length helps focus criterion generation on substantive differences.

  • Evaluation connects rubric quality to training outcomes. The paper tests both preference-ranking accuracy and downstream improvements after GRPO training. Its Positive-only ablation shares the same training and scoring pipeline but removes the near-miss and comparison stages, providing a useful comparison for assessing the contribution of contrastive anchoring.

Critical weaknesses
  • Limited validation of the synthetic contrasts. The same frozen model generates both the pseudo-reference and its near-miss, without independently checking their quality. Although the paper reports better reward discrimination, it does not directly establish whether the injected errors reflect common, important failures in actual policy rollouts. The benchmark gains are encouraging, but the proposed mechanism remains insufficiently validated.
  • Reliance on a single evaluation judge. All downstream policy evaluations use GPT-5.6 Luna, including both benchmark scoring and pairwise comparisons. Agreement between these evaluation formats therefore does not establish robustness across judges. Independent human assessment or additional judges would strengthen the evidence for genuine quality improvements.
  • CARE adds near-miss generation and criterion extraction to the Positive-only pipeline, requiring extra inference at every rollout group. Without a compute-matched baseline, it is difficult to determine how much of the improvement comes from contrastive anchoring rather than the additional feedback-generation budget.
Questions for authors
  • How do you verify that the generated near-misses are genuinely worse than the pseudo-references and reflect important errors in actual policy rollouts?
  • Have you compared CARE with a compute-matched Positive-only baseline to separate the benefit of contrastive anchoring from that of additional inference?
1 comment
Reviewer_S5dDOpenReview ↗

Thank you for your careful comments

Since you have claimed some of the issues, i decide to increase my score. I find the additional experimental results convincing.

Reviewer_KpzvOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper works on synthetic rubric generation. They use the same policy model as the generator and generate targeted “near miss” samples that are used to generate rubrics that capture differences and are comprehensive. The authors show these rubrics are better in preference ranking and also downstream policy training.

Critical strengths

The idea is clear and well-motivated supported by experiments in both preference capturing (RM) and downstream policy. I particularly like that they use the same frozen policy as the rubrics generator, unlike most papers in the field that use some stronger SOTA model. Results across multiple benchmarks and models along with ablations and analysis are provided.

Critical weaknesses

i think some comparison with other synthetic rubrics generator baselines (e.g., OnlineRubric and CaT as cited and many other existing methods) would help strengthen the paper and better understand the significance of the contribution.

Also, the authors state that their motivation is to stay in the policy’s decision boundary and be on-policy. However, to the best of my understanding, they use the positive samples and artificially corrupt them to get negative samples. This means that negative samples would not necessarily be something that the policy generates naturally, and therefore is not on-policy.

Less significant, but naturally this method requires more compute in generation. An analysis on the performance vs. cost would help better understand whether the extra generation cost is justified by the performance improvements.

Questions for authors

none

Reviewer_YTo6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper works on rubric-guided policy optimization. The authors propose to use the same model use for policy alone to construct adaptive rubrics that can distinguish different responses. At each step, the authors use the model to construct a pseudo-positive reference by aggregating multiple responses from the running policy, then "edit" the reference into a negative one. By comparing the two, the model propose a new rubric. Finally, the authors score these responses based on the new rubric, and conduct GRPO training. The proposed method is evaluated on two benchmarks using three models.

Critical strengths
  1. The problem studied in this paper is important.
  2. The paper is in general well-written and easy to follow.
Critical weaknesses
  1. My first main concern is the soundness and novelty of the proposed method. First, using different responses (e.g., majority voting) for better RL is known [1], and aggregating different responses beyond majority voting can also lead to better quality response has also been established [2]. Given the advantage of [1, 2], It is unclear whether the proposed rubric-based RL pipeline offers advantages over directly distilling the synthesized answers into the policy that are simpler: the repeated construction and evolvement of prompt-specific rubrics lead to notable computational overhead, but they are only used to train the policy once. This raises the question about whether the additional cost does benefit the training. Also, it is unclear to me why we should use a synthetic near-miss response for comparison; it can be better than any individual on-policy responses. As a result, the proposed rubric may be suboptimal, which can focus on the "gap" between the two synthesized responses, but not fully capture the quality of the actual generated one.

[1] TTRL: Test-Time Reinforcement Learning.

[2] LLMs Can Generate a Better Answer by Aggregating Their Own Responses.

  1. My second concern is the incomplete discussion about the existing work on rubric generation, rubric-based reward modeling, and rubric reliability [1–7]. Moreover, I also notice that some closely related work on aggregation-based rubric construction were not discussed [8-10]. This is just an incomplete list of some papers I know by chance. I believe more related work exist.

[1] OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignment.

[2] AutoRubric: Rubric-Based Generative Rewards for Faithful Multimodal Reasoning

[3] CARMO: Dynamic Criteria Generation for Context Aware Reward Modelling

[4] LLM-Rubric: A Multidimensional, Calibrated Approach to Automated Evaluation of Natural Language Texts.

[5] C2: Scalable Rubric-Augmented Reward Modeling from Binary Preferences

[6] RubricBench: Aligning Model-Generated Rubrics with Human Standards

[7] Auto-Rubric: Learning From Implicit Weights to Explicit Rubrics for Reward Modeling

[8] LLMs Can Generate a Better Answer by Aggregating Their Own Responses.

[9] RubiCap: Rubric-Guided Reinforcement Learning for Dense Image Captioning.

[10] CDRRM: Contrast-Driven Rubric Generation for Reliable and Interpretable Reward Modeling.

  1. My third concern is that the experimental validation is insufficient. On one hand, the above aggregating based (non-rubric based version) should be compared in my opinion to justify the necessity for rubric. Moreover, I noticed that the authors already discussed some related papers, such as CaT、OnlineRubrics、EvoRubric、EvoRubrics, but not compare with them empirically. Also, HealthBench and ResearchQA benchmarks are both knowledge-intensive QA tasks. The generalizability to broader open-ended tasks is also missing.
Questions for authors

Please see my comments above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.