acceptodds
Under review as a conference paper at ICLR 2027

RubricEvo: Sustaining Learning Signals via Evolving Graph-based Multi-Level Rubrics in Open-Ended Reinforcement Learning

Abstract

Rubric-based reinforcement learning provides a practical way to optimize open-ended tasks without verifiable answers, yet fixed, flat rubrics become increasingly ineffective as the language-model policy evolves. Once broad criteria are mastered, rubric scores saturate and reward gaps among on-policy responses diminish; simultaneously, optimizing against a single global checklist favors a narrow set of rubric-compatible behaviors and suppresses alternative valid strategies. We propose , which represents each query-specific rubric as an evolving, multi-level graph. The graph organizes rubric into fixed nodes that preserve fundamental requirements, nodes that capture major quality dimensions, and nodes that provide fine-grained, strategy-dependent guidance. While Principle nodes remain unchanged, Aspect and Specific nodes can be replaced or branched as the policy evolves. During training, RubricEvo measures reward resolution and semantic diversity over on-policy response groups to guide the evolution of the graph-based rubric. Experiments across multiple domains and Qwen3-1.7B/4B/8B policies show that RubricEvo improves downstream performance while preserving more discriminative rewards and greater high-quality output diversity. All datasets and code are available at https://anonymous.4open.science/r/RubricEvo/.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Reinforcement Learning, Rubric-Based Evaluation, Open-Ended Generation, Dynamic Evaluators, Response Diversity

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_A57DOpenReview ↗
Rating
3(+2)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2(+1)
Read review
Paper summary

RubricEvo organizes rubrics into required criteria and alternative strategies, then evolves the graph when reward resolution or response diversity deteriorates. Validation and rollback mechanisms aim to maintain reward quality during evolution.

Critical strengths

Table 2 reports improvements across all 15 model–benchmark combinations. Appendix A.5 provides positive paired instance-bootstrap confidence intervals for every comparison. Table 3 directly tests graph structure while retaining online evolution. On Qwen3-1.7B Writing, the full method exceeds the evolving flat-rubric ablation by 1.58 points on WritingBench and 0.77 on Creative Writing. Blind human ranking preferences and WritingBench evaluation using a separate Qwen-7B critic provide supporting evidence beyond GPT-4o-based evaluation.

Critical weaknesses

The graph contribution is established in a limited setting. The controlled ablation covers one model size and domain. In the main comparison, EvoRubrics uses LoRA policy/generator adapters, whereas RubricEvo uses full-parameter policy training and an external GPT-4o evolver. These differences limit attribution of the broader gains specifically to graph structure.

GPT-4o both evolves rubrics and evaluates four of five benchmarks, creating possible shared-preference bias. Existing human and alternative-judge evidence mitigates, but does not eliminate, this concern.

BGE-M3 underlies both evolution diagnostics and diversity evaluation; the effective-mode analysis also uses these embeddings. An independent diversity measure would strengthen the claim that the method preserves distinct valid strategies.

Only one training seed is reported. Instance-bootstrap intervals do not quantify optimization variability. Numerical evolution-trigger thresholds, persistence settings, and rollback tolerances also require fuller specification.

Questions for authors
  • Does the existing evolving-flat ablation reproduce its advantage in another domain or model size?
  • Can the authors specify the evolution hyperparameters and quantify training-seed variability?
  • Do the diversity conclusions hold under an independent representation or human assessment of distinct strategies?
7 comments

Overall response. Thank you for your thoughtful feedback. To our knowledge, RubricEvo is the first work to jointly motivate rubric evolution by declining reward resolution and strategy diversity in open-ended RL. Its rubrics evolve alongside the policy as graphs that preserve shared requirements and give different valid strategies their own branches. We would be grateful for your reconsideration the score in light of the results below.

Q: How should we interpret the EvoRubrics comparison?

R: EvoRubrics’ dual-LoRA architecture is central to how its policy and rubric generator co-evolve [1]. We retained it to reproduce the baseline faithfully and compare the two methods fairly, with each retaining its defining technical choices.

Q: Reuse of GPT-4o

R: Claude Haiku 4.5 re-scored the same saved responses (Table R1).

Table R1. Claude Haiku 4.5 re-evaluation.

BackboneMethodHealthBenchLMB-ENCreative Writing v3WildBench v2
Qwen3-1.7BBase Model12.8413.3540.7944.52
RaR17.7216.8642.0345.73
EvoRubrics18.9117.4840.9647.18
RubricEvo (Ours)20.6319.2741.4146.93
Qwen3-4BBase Model23.5722.8146.7253.28
RaR28.8424.1349.3555.47
EvoRubrics31.1626.4248.7656.83
RubricEvo (Ours)32.4728.0850.6257.41
Qwen3-8BBase Model31.2630.7453.1861.23
RaR37.8335.9256.4764.18
EvoRubrics39.1438.7657.8265.74
RubricEvo (Ours)41.6840.5357.3666.32

Although absolute scores shift systematically across judges, RubricEvo still achieves the best overall results under Claude Haiku 4.5.

Q: Does the diversity finding depend on BGE-M3?

R: Using 16 responses per prompt, we measure semantic dispersion with Qwen3-Embedding-8B. Table shows how the results vary across metrics and benchmarks. The results below show that RubricEvo still achieves the highest diversity score when evaluated with a newer, larger embedding model than the one used during training.

BenchmarkBase ModelRaRStatic Graph-RLEvoRubricsRubricEvo (Ours)
HealthBench1.0000.8020.8791.0581.092
LMB-EN1.0000.7810.8661.0491.076
WritingBench1.0000.7660.8521.0741.105
Creative Writing1.0000.8140.8971.0671.094
WildBench1.0000.7930.8811.0561.048
Average1.0000.7910.8751.0611.083

Q: Does the graph advantage extend across scales, domains, and seeds?

R: The original ablation used a fixed model size and domain, as in prior work [2, 3]. We extended the comparison to Qwen3-4B and Qwen3-8B on WritingBench and to Chat and Medical tasks at 1.7B. We use the official 7B WritingBench Critic for WritingBench and the same Claude Haiku 4.5 judge as above for all other benchmarks. Table R3 reports mean ± standard deviation across training seeds 42, 43, and 44. RubricEvo achieves higher mean scores than Evolving Flat-RL across the tested model sizes and domains.

Table R3. Controlled comparison across model sizes and domains.

BackboneDomainBenchmarkEvolving Flat-RLRubricEvo (Ours)
Qwen3-4BWritingWritingBench76.74 ± 0.5878.46 ± 0.43
Qwen3-8BWritingWritingBench80.12 ± 0.4781.17 ± 0.36
Qwen3-1.7BChatWildBench v246.31 ± 0.4246.87 ± 0.34
Qwen3-1.7BMedicalHealthBench18.91 ± 0.7620.48 ± 0.61
Qwen3-1.7BMedicalLMB-EN18.36 ± 0.5319.41 ± 0.46

Q: What are the key evolution hyperparameters?

R: The appendix (Table 7) lists the training hyperparameters. Here we provide the key evolution settings. The reward-gap and diversity trigger thresholds are 0.85 and 0.95 of their initial values, with a quality floor of 0.55 and a persistence requirement of two consecutive audit windows. Rollback checks use a minimum retention of 50% of the predicted reward-gap gain and a maximum shadow-quality drop of 0.05. For diversity edits, mean diversity over two probation windows must improve over its pre-edit value by at least 0.5% in relative terms and 0.001 in absolute terms. We originally left these settings in the accompanying code and have now documented them explicitly in the revised appendix.

References

[1] EvoRubrics: Dynamic Rubrics as Rewards via Adversarial Co-Evolution for LLM Reinforcement Learning. §4.1, 2026.

[2] MA-RLHF: Reinforcement Learning from Human Feedback with Macro Actions. §4.3.1, ICLR 2025.

[3] DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. §5.2.1, 2024.

We would be grateful if you would reconsider your assessment in light of the additional evidence, and we would be happy to address any further questions or concerns.

Thank you for the additional experiments. They broaden the evidence, but two central issues remain. First, please specify how Evolving Flat-RL is matched to RubricEvo in policy training, evolver, evolution opportunities, and validation budget; this determines whether Table R3 isolates graph structure. Second, the independent embedding evaluation demonstrates dispersion under another representation, but does not establish preservation of distinct valid strategies. The average advantage over EvoRubrics is modest (1.083 versus 1.061), lacks uncertainty, and reverses on WildBench. Please either provide direct support for the strategy-preservation interpretation or narrow that claim. The Claude results mitigate evaluator dependence, but their three ranking reversals and missing paired uncertainty also warrant more qualified performance claims.

I would appreciate additional evidence addressing these concerns before finalizing my scores.

Reviewer_DfTvOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

Basic idea: update the question-specific rubrics used for questions as RL progresses. The extra twist is some heuristic to provide rubrics that allow multiple possible solutions.

The writing is generally poor and the ideas badly explained. For example, the introduction is full of undefined jargon (“smoothed reward gap”, “on policy diversity” in the first paragraph alone!). Related: the paper is full of apparently arbitrary decisions (e.g., why measure diversity with cossim of embeddings of responses? Particularly since LLM judges are being used anyway to evaluate semantics. Also, why constrain the replace/branch pipeline instead of just letting the LLM update the rubric as it sees fit?). These are not adequately motivated (or ablated post-hoc) to justify. The method seems to work fine, but not so much that I could imagine ever actually using it instead of, e.g., just asking a current LLM to design an entirely new pipeline from scratch.

Critical strengths

see summary

Critical weaknesses

see summary

Questions for authors

none

1 comment

Dear Reviewer DfTv,

As the first author, I appreciate the time and effort involved in reviewing. However, I consider this review entirely irresponsible in its present form. Its single paragraph makes broad criticisms without substantively engaging with the paper.

First, I consider the descriptions in the introduction straightforward and readily understandable. More importantly, the relevant concepts are formally defined in Section 4.1. Calling these terms vague without engaging with those definitions raises serious concerns about whether the manuscript received a sufficiently careful reading.

Second, embedding-based cosine similarity or distance is an established approach to evaluating semantic diversity [1–4], including in Artificial Hivemind, a NeurIPS 2025 Best Paper [2]. Lexical metrics such as Distinct-n and Self-BLEU measure surface variation, which need not reflect differences in meaning. Our use of BGE-M3 follows this established embedding-based approach. These precedents do not make the metric immune to criticism, but dismissing it as arbitrary without identifying a concrete problem with its use here is not a substantive technical assessment.

Third, I regard the suggestion to let the LLM freely modify rubrics as not even wrong in its present form: it bypasses the methodological question of how reward updates should be controlled and validated. REPLACE/BRANCH operations and validation serve precisely this purpose. The review does not explain why removing these constraints would constitute a sounder design.

I welcome specific technical criticism and remain willing to clarify our choices. However, I cannot accept this review as an adequate assessment of our work. I would welcome any further details you could share about your assessment and would be happy to address any questions or concerns. I would also be grateful if you would reconsider your score.

References

[1] Robert Kirk et al. Understanding the Effects of RLHF on LLM Generalisation and Diversity. ICLR, 2024.

[2] Liwei Jiang et al. Artificial Hivemind: The Open-Ended Homogeneity of Language Models (and Beyond). NeurIPS, 2025. Best Paper Award.

[3] Yanzhu Guo, Guokan Shang, and Chloé Clavel. Benchmarking Linguistic Diversity of Large Language Models. TACL, 2025.

[4] Guy Tevet and Jonathan Berant. Evaluating the Evaluation of Diversity in Natural Language Generation. EACL, 2021.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.