acceptodds
Under review as a conference paper at ICLR 2027

RARE: Reinforcement Learning with Adaptive Rubric Evolution for Open-Ended Generation

Abstract

Verifiable rewards have advanced reinforcement learning in mathematics and code. However, many valuable open-ended tasks share three properties: no automatic verifier of overall quality, multiple requirements to satisfy jointly, and multiple valid responses to each query. Training therefore needs feedback on requirement satisfaction across different valid responses. Rubric-based reinforcement learning provides this feedback by aggregating scores from explicit criteria. As learning progresses unevenly, some criteria become fully satisfied and lose discrimination while others remain informative. Dynamic methods revise shared rubrics or select evolving criteria to restore discrimination, but may also change useful criteria. To address this, we propose Reinforcement Learning with Adaptive Rubric Evolution (RARE). The core idea is to use current policy feedback to guide both which criteria change and what their replacements assess. Sample-level updates replace criteria fully satisfied across current outputs with checks of remaining quality differences, retaining all other criteria and scores. Within this rule, we compare rewarding desired qualities with penalizing observed flaws. Video script generation exemplifies these properties and supports content production. Its demands for coherent, executable narratives under user constraints make it a challenging testbed. Targeting flaws performs best across three video types with a mean gain of 3.82 points over matched static rubrics. Model and human preferences support these gains. Improvements also extend to writing and general generation benchmarks.

Primary area:
foundation or frontier models, including LLMs
Keywords:
reinforcement learning, large language models, rubric-based rewards, adaptive reward design, open-ended generation, script generation

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_BHgiOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper examines methods for turning non-verifiable problems (such as video script rating) into verifiable problems, by using an external model (GPT5.4) to generate a rubric to score the outputs. In this case the rubric evolves as particular criteria saturate, allowing the model to continually improve on distinct dimensions. This is quite simple and neat; the authors run extensive experiments with appropriate baselines, comparing criterion variants that exhibit desirable properties or avoid flaws (the latter "negative" works best). They also Claude / GPT as raters and find that negative beats sample static (a control in without criterion evolution) quite handily; human raters provide a similar result.

I thought this was quite a neat paper. It's a simple, clear innovation that is nicely explored, well tested, and clearly reported.

Critical strengths

Very simple idea; nicely implemented. Elegant experiments, clearly reported.

Critical weaknesses

In this Rubric evaluation literature I struggle with the question of circularity; for example here GPT5.4 is the rubric evaluators, as well as (one of) the eventual judge(s). Nice to have human validation, but what the actual ground truth here?

Questions for authors

Does this work for domains other than the one tested?

Reviewer_AXy1OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RARE, a framework for evolving rubric-based rewards in reinforcement learning for open-ended generation. The method identifies criteria that receive full scores across all current policy rollouts and replaces them with newly generated criteria targeting remaining quality differences, while preserving unsaturated criteria. The authors investigate different replacement strategies and task-level rubric updates.

The authors primarily conduct experiments on video script generation across three production styles using Qwen3-4B. The best-performing Negative variant improves the average score by 3.82 points over static rubrics. The paper also reports comparisons with existing evolving-rubric methods, criterion-level analyses, human preference evaluations, and additional results on WritingBench and BiGGen Bench.

Critical strengths

The paper identifies a concrete limitation of static rubric-based rewards and proposes a straightforward criterion replacement mechanism. The empirical results suggest that targeting observed flaws can provide more discriminative feedback than introducing positive quality criteria.

Critical weaknesses
  1. Limited methodological novelty and insufficient isolation of the proposed mechanism.

The central contribution is essentially a heuristic: replace criteria once all sampled responses receive full scores. While this is a reasonable engineering choice, prior work has already explored evolving rubrics based on policy outputs and score discrimination.

More importantly, the experiments do not adequately isolate the benefit of saturation-based selective replacement from the general benefit of repeatedly introducing new, flaw-focused criteria. The comparisons with existing evolving-rubric methods are useful, but these methods differ in several aspects beyond the proposed selection rule. Consequently, the evidence does not convincingly establish this heuristic's specific contribution.

  1. Insufficient consideration of rubric validity and quality.

A fundamental challenge in rubric-based RL for open-ended tasks is constructing criteria that meaningfully reflect task quality. Automatically generated criteria may be incomplete, redundant, or based on incidental differences among sampled responses.

RARE primarily addresses score saturation and discrimination, but greater reward variance does not necessarily imply more meaningful or better-aligned supervision. Replacing saturated criteria with new ones does not itself ensure that the resulting rubrics cover important quality dimensions or avoid spurious criteria.

Although the authors include prompting constraints for non-redundancy and report criterion-level analyses, the paper provides limited independent evidence that the evolving criteria consistently reflect meaningful quality improvements. The reliance on powerful external models for rubric generation and judging further makes it important to distinguish the contribution of the proposed algorithm from that of the generated feedback.

Questions for authors
  1. Can the authors isolate the contribution of saturation-based replacement through a compute-matched comparison with alternative selection rules, such as random replacement or variance-based selection, while keeping the rubric generator and update budget fixed?

  2. What evidence supports the validity and importance of newly generated criteria beyond their ability to produce score differences? In particular, how does the method prevent redundant or incidental criteria from becoming reward signals?

4 comments
AuthorsOpenReview ↗

Follow-up: Additional Evidence for RARE’s Criterion Selection and Adaptation

Dear Reviewer AXy1,

We would like to follow up on our earlier message and highlight two findings that strengthen the evidence for RARE’s design choices:

  1. Full-score saturation outperforms alternative selection rules under matched update and compute budgets. In the controlled comparison on the 3D / Stop-motion domain, averaged over two training seeds, RARE outperforms zero-variance and random selection by 6.24 and 7.46 points, respectively. These comparisons isolate the contribution of the selection rule: full-score saturation replaces criteria that all current responses already satisfy, while preserving criteria that capture requirements the policy has yet to master.

  2. New criteria provide useful feedback beyond the triggering batch, while the policy largely retains previously satisfied requirements. Our concrete example illustrates how a new criterion captures observable, request-relevant quality differences distinct from those assessed by retained criteria. Across 940 first-round update events, the new criteria still distinguish later-checkpoint responses in 43.0% of events at the end of training. Meanwhile, 90.4% of newly sampled rollouts still receive full marks on the replaced criteria, supporting the value of introducing new feedback while preserving previously learned behavior.

Together, these results support RARE’s central contribution: selectively replacing fully satisfied criteria with new, informative criteria allows training feedback to adapt as the policy improves.

We would appreciate your assessment of this additional evidence. If it addresses your concerns, we would be grateful if you could consider revising your score upward to reflect the strengthened empirical support for our contribution. Please let us know if you have any remaining concerns so that we can address them during the discussion period.

Thank you again for your time and consideration.

Best regards,
The Authors

Thank you for the detailed response and additional experiments.

However, my main remaining concern is whether the newly generated criteria actually lead to targeted improvements in the specific quality dimensions they are designed to address. While the results demonstrate improved overall performance and criterion relevance, they do not directly establish whether the policy learns to correct these specific flaws rather than primarily adapting to general judge preferences.

AuthorsOpenReview ↗

Follow-up: Targeted improvements in the quality dimensions addressed by new criteria

Thank you for acknowledging our detailed response and additional experiments. We added longitudinal and paired analyses to examine whether the policy improves on the specific quality dimensions targeted by new criteria. The results show targeted improvements, supported by an independent judge and fixed evaluation metrics.

1. Quality measured by the new criteria improves throughout subsequent training.

We tracked 940 Negative update events from the first training epoch, covering 1,210 pairs of new and old criteria. For each query, we rescored the current eight rollouts immediately after replacement. We then sampled eight new responses at the end of the first epoch and at the final checkpoint.

JudgeImmediately after replacementEnd of first epochEnd of trainingImmediate-to-final gain95% CI
Qwen3.7-Plus0.8420.9060.931+0.089[0.079, 0.099]
Claude Sonnet 50.8380.8990.925+0.087[0.076, 0.098]

Qwen3.7-Plus is the training judge. Claude Sonnet 5 serves as an independent evaluator and participates in neither criterion generation nor training scoring. Both judges observe steadily increasing scores on the new criteria.

The immediate scores in this table are computed over the 940 update events tracked through later checkpoints. The paper's average post-switch score of 0.82 covers all 3,336 Negative update events. These statistics refer to their respective event sets.

2. RARE's advantage is concentrated in the qualities targeted by the new criteria.

For the same 940 queries, we sampled eight new responses from each of the final RARE and Sample-Static checkpoints. Both judges evaluated these responses using the new criteria and their corresponding old criteria, with the training method hidden.

JudgeRARE − Static on new criteriaRARE − Static on old criteriaDifference in gains95% CI
Qwen3.7-Plus+0.052+0.008+0.044[0.031, 0.057]
Claude Sonnet 5+0.049+0.007+0.042[0.028, 0.055]

The difference in gains is the RARE–Static advantage on new criteria minus its advantage on old criteria. RARE shows a larger improvement on the new criteria while maintaining performance on the original requirements. Claude Sonnet 5 finds the same pattern, providing independent support for improvement in the targeted qualities.

3. Fixed evaluation metrics show the same pattern of targeted improvement.

We mapped 5,273 new Negative criteria to the four fixed evaluation dimensions. The mapping used only the query, the new and old criteria, and the learning direction. Final evaluation scores were excluded from this step. For each target dimension, we selected the queries that had generated criteria targeting that dimension. We then compared the gains in the target dimension with the average gains in the other dimensions.

Target dimension of new criteriaNumber of criteriaGain in target dimensionAverage gain in other dimensions
Query Fulfillment1,514+4.82+3.19
Narrative Coherence749+5.86+3.21
Cinematic Executability1,497+4.61+3.28
Dramatic Effectiveness1,513+4.27+3.31
Overall5,273+4.72+3.25

The average gain is 4.72 points in the target dimensions and 3.25 points in the other dimensions. The difference is 1.47 points, with a 95% confidence interval of [1.11, 1.84].

This analysis uses query subsets associated with each type of new criterion. Figure 4(b) reports gains over the full test set: +4.00 for Query Fulfillment, +4.45 for Narrative Coherence, +3.78 for Cinematic Executability, and +3.07 for Dramatic Effectiveness. The two analyses describe performance at their respective aggregation levels.

4. The evolving rubric guides policy learning through rewards.

The dynamic rubric is used exclusively for reward computation and remains outside the policy's input. It influences the policy through reward-based updates to the generated responses. The longitudinal analysis shows that later responses receive higher scores on the new criteria. The paired comparison locates the larger RARE–Static advantage in those criteria. Fixed evaluation metrics further show that gains are concentrated in the corresponding quality dimensions.

Together, these results support targeted learning of the improvements specified by the new criteria. The agreement between the training judge, the independent judge, and the fixed evaluation metrics provides consistent evidence for this conclusion.

AuthorsOpenReview ↗

A Kind Reminder Before the Rebuttal Deadline

Dear Reviewer AXy1,

With the rebuttal deadline approaching today, we would greatly appreciate it if you could take a look at our responses and share any remaining concerns. We would be happy to provide further clarification while there is still time to respond.

Thank you again for your time and valuable feedback.

Best regards, The Authors

Reviewer_LRweOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes RARE to address reward saturation in rubric-based reinforcement learning for open-ended tasks. In my opinion, its main contribution is its selective rubric update procedure. The proposed method replaces only the fully saturated criteria and uses differences in current outputs to generate new criteria. The procedure also preserves the other criteria and their scores and keeps the rubric size and max total score unchanged.

The paper evaluates RARE on video script generation tasks across three production styles as well as additional experiments on writing and text generation. It finds that RARE improves over fixed-rubric baseline and the existing dynamic rubric methods, MPO and RLER, on video script generation benchmarks. They also show that human assessments favor its best-performing RARE variant over fixed-rubric method on video script generation.

Critical strengths

This paper tested and showed that RARE’s improvements are noticeable to human readers. In addition to evaluation against fixed-rubric and existing dynamic-rubric baselines, this paper also conducted a blinded human assessment to compare RARE with the fixed-rubric baseline, Sample-Static, in video script generation.

Critical weaknesses
  1. The main distinction and novelty of this paper compared to existing methods is its selective rubric update procedure: it (1) replaces only fully satisfied criteria, and uses differences in current outputs to generate new criteria for replacement, (2) keeps the other criteria and their scores, and keeps the rubric size and max scores unchanged. However, their evaluations do not isolate the benefits of these design choices. For example, is full-score saturation a better replacement criterion than zero variance? In the evaluation, the RLER comparison changes several design choices together: RLER permanently keeps the original criteria and generates additional criteria that are then filtered by score variance. The task-level MPO comparison evaluates a different shared-rubric revision procedure. These evaluations support the performance of RARE but still leave it unclear whether RARE’s particular design choices account for this improvement.

  2. Robustness concern. The main results of the evaluation are averaged over four training seeds but variability across the seeds is not reported, which makes it difficult to assess how consistent the improvements are. Also the additional text task experiments use one seed and show only small gains over baselines, which makes it unclear whether the gains reflect improvements of RARE or training variation. Within each task (video script generation vs text task), the experiments also use the same policy model (Qwen3-4B), rubric generator (GPT-5.4), and training judge (Qwen3.7-Plus for script generation and GPT-4.1-mini for the additional text tasks). Whether their improvements over the baseline are consistent when any of these models is changed is unclear. It will strengthen the evidence for robustness if they can report per-seed results and repeat the baseline comparisons with alternative model choices on the same task.

Questions for authors
  1. Have the authors considered running ablations over their selective rubric update procedure (1. replaces only fully satisfied criteria, 2. generation of replacements from differences among current outputs, 3. preservation of the remaining criteria and their assigned scores) to disentangle and clarify how each choice contributes to the improvement?
  2. How much do RARE’s improvements over the baselines vary across the four training seeds, and do they support the reported advantage?
  3. For the additional text tasks, is there evidence that the small improvements reflect a improvement of RARE rather than training variation?
Reviewer_WsmHOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes RARE, a rubric-based reinforcement learning framework for open-ended generation. At the sample level, RARE replaces criteria that receive full scores across all current responses, while retaining the remaining criteria and their scores. Replacement criteria are generated from differences between the sampled responses. Experiments primarily evaluate video-script generation, with additional evaluations on WritingBench and BiGGen Bench. Flaw-focused replacements perform best, improving script quality over matched static rubrics.

Critical strengths
  1. The method addresses a concrete limitation of rubric-based RL: individual criteria can become uninformative as the policy improves. Selectively replacing saturated criteria provides a clear mechanism for restoring reward discrimination while preserving other feedback.

  2. The evaluation includes comparisons across replacement strategies, analyses of criterion discrimination and retention, and both model and human preference judgments. These provide complementary evidence beyond the primary pointwise scores.

Critical weaknesses
  1. The validity and task dependence of the evolving rubrics remain insufficiently established. Open-ended tasks do not require a unique reference answer, but their evaluation criteria still need to reliably represent user-valued quality. Restoring score variance does not necessarily mean that a replacement criterion captures a meaningful improvement; it could introduce stylistic preferences or additional requirements not implied by the query. GPT-5.4 generates the revisions and also provides the primary final evaluation, creating a possible source of shared preferences. The independent judge and human study are useful, but they do not directly validate the replacement criteria themselves. Could the authors report independent human assessments of criterion relevance, consistency with the original request, and robustness to alternative rubric generators?

  2. The practical significance and generality of the gains need stronger support. The main script improvement is 3.82 points on a normalized 100-point scale, while the additional-task gains are smaller: 8.26 to 8.34 on WritingBench and 4.152 to 4.259 on BiGGen Bench. The human preference results are encouraging, so these improvements should not simply be dismissed. Nevertheless, uncertainty estimates for the main score differences and multiple training seeds for the additional tasks would help establish whether the benefits are reliable beyond the primary setting.

Questions for authors

See Weakness

2 comments
AuthorsOpenReview ↗

Follow-up: Additional Evidence for RARE’s Effectiveness and Robustness

Dear Reviewer WsmH,

We would like to follow up on our earlier message and draw your attention to two findings in our additional results that we believe strengthen the case for RARE:

  1. RARE’s gains persist beyond the original criterion generator. With an alternative generator, RARE still outperforms the static baseline by 3.24–3.64 points across all three video domains. Together with human and independent-model audits of 180 criteria supporting their task relevance and request alignment, these results strengthen the evidence that the gains come from the criterion-update mechanism, beyond the particular generator used.

  2. The improvements are consistent across runs and tasks, and are reflected in human preferences. The 95% bootstrap confidence intervals for the video-task gains are above zero in every domain, and RARE outperforms Static in all three training seeds on both WritingBench and BiGGen Bench. In the human evaluation, RARE was preferred in 38 pairs versus 13 for the static baseline, with 9 ties or unresolved cases, supporting the practical value of the improvements.

These findings support our central contribution: adapting training criteria as the policy improves provides more effective feedback than keeping the criteria fixed.

We would appreciate your assessment of this additional evidence. If it addresses your concerns, we would be grateful if you could consider revising your score upward to reflect the strengthened empirical support for our contribution. Please let us know if you have any remaining concerns so that we can address them during the discussion period.

Thank you again for your time and consideration.

Best regards,
The Authors

AuthorsOpenReview ↗

Follow-up on Our Rebuttal

Dear Reviewer WsmH,

With the rebuttal deadline approaching today, we would greatly appreciate it if you could take a look at our responses and share any remaining concerns. We would be happy to provide further clarification while there is still time to respond.

Thank you again for your time and valuable feedback.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.