acceptodds
Under review as a conference paper at ICLR 2027

Co-Evolving LLM Evaluators and Policies via DynamicRubric

Abstract

Post-training with evaluator feedback on policy-induced samples serves as a major mechanism for improving large language models. As policies improve, these sampled responses become close in quality. These close responses create a bottleneck for policy optimization: when evaluator score gaps collapse or reverse, policy supervision becomes weak or misleading. We theoretically characterize why these gaps matter through a probability allocation view: shifting probability mass from one response to another yields a directional gain exactly equal to the evaluator score gap between them. This identifies relative score gaps as the policy optimization signals that guide updates. We therefore propose DynamicRubric to co-evolve evaluators and policies using response-set-conditioned rubrics. An LLM can take either role and self-evolve by learning to supervise its own updates without an external reward model or judge. Using an 8B backbone, DynamicRubric produces an evaluator that is competitive with a 70B reward model on preference evaluation. On open-ended tasks, DynamicRubric provides stronger policy supervision than static rubrics generated by a 235B model, and performance gains generalize to verifiable tasks. A DynamicRubric-optimized model is fully deployed for AI answering in a large-scale production environment, handling tens of millions of requests per day and improving key online metrics. These results suggest a principle for post-training: evaluators should evolve with the policies they supervise.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Bilevel Optimization, Rubric-Based Supervision

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_jmmZOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes DynamicRubric, a framework that co-evolves LLM evaluators and policies using response-set-conditioned rubrics. A rubric generator produces weighted binary evaluation criteria and is optimized using discriminability and anchor-ranking rewards. The generator and policy are alternately updated with GRPO. Experiments demonstrate competitive evaluator performance against larger models and improvements in open-ended generation, verifiable tasks, and production deployment.

Critical strengths
  1. Well-motivated approach: Dynamically adapting evaluation criteria to current-policy responses is an intuitive and promising approach to improving reward supervision.
  2. Strong empirical results: The trained 8B rubric generator achieves competitive performance against substantially larger reward models, while the resulting policy outperforms strong baselines across multiple benchmarks.
  3. Evidence for co-evolution: Controlled experiments demonstrate that updating evaluators using current-policy responses provides better supervision than using stale evaluators.
  4. Broad evaluation: The improvements are validated across multiple benchmarks, model backbones, verifiers, and LLM judges.
Critical weaknesses
  1. Limited theoretical justification: The theoretical results largely follow from standard properties of linear objectives and function-class inclusion. Moreover, the emphasis on raw score gaps is not fully reconciled with GRPO's reward normalization, and the theoretical bound relies on an additional calibration assumption.
  2. Insufficient baseline comparisons: The experiments do not fully isolate the contributions of response-set conditioning and generator training. Comparisons with closely related co-evolution methods, such as EvoRubric and EvoRubrics, are also missing.
  3. Limited analysis of discriminability: The reported accuracy improvements may partly reflect reduced score ties rather than improved ranking quality. This concern is particularly relevant because listwise nDCG improvements are relatively modest.
  4. Limited analysis of robustness and stability: Potential reward hacking, response-length biases, and long-term co-evolution stability are not sufficiently investigated. The co-evolution experiments cover only two rounds.
Questions for authors
  1. Scaling: Have the authors explored training larger rubric generators (e.g., 32B)? Does DynamicRubric continue to improve performance as model size increases?
  2. Baseline comparisons: Can the authors compare against a trained prompt-only rubric generator under matched conditions and explain the differences from EvoRubric and EvoRubrics?
  3. Score ties: How much of the evaluator accuracy improvement comes from reducing tied scores? Can the authors report tie rates and clarify the tie-handling protocols in Tables 1 and 6?
  4. Reward reliability: Can the authors provide further evidence connecting evaluator discriminability to downstream policy improvements and investigate potential reward hacking or response-length biases?
  5. Data contamination: Have the authors checked for overlap between the Nectar training data and the evaluation benchmarks, particularly UltraFeedback?
  6. Long-term stability: How does performance evolve beyond two co-evolution rounds? Does it continue to improve or eventually saturate?
3 comments

We thank Reviewer jmmZ for the valuable feedback. Here are our responses.

W1. Theory

(1) Our theory explains how LLM policies and evaluators should co-evolve during post training. Other co-evolution methods lack this theoretical analysis. We explain why evaluators fail as policies improve and derive a solution (Section 3). Our experiments show that this solution works (Section 4).

(2) Our work studies evaluator failures as policies improve during post training. The evaluator can score worse responses higher than better ones. GRPO cannot correct these errors through score normalization. We study this phenomenon in common post training paradigms and use GRPO only as the specific RL algorithm in our implementation.

W2/Q2. Comparisons

(1) We have already isolated the effects of response access and continued training in the initially submitted version.

  • Response access. Our DR-Generator-8B (51.2 on AEOLLM, for example) outperforms the strong Qwen3-32B baseline (29.7) on all six accuracy benchmarks. We also compare the strong Nemotron-70B baseline with and without responses (UltraFeedback 72.8 → 77.4, Personalized-RB 68.6 → 61.9) (Table 1). Simply seeing responses does not guarantee better evaluation.

  • Continued training. Figure 2(a) starts from the same generator and continues training on either old or current policy responses. Performance stays flat or drops with old responses, while seeing current responses brings further gains.

(2) We have already compared these related works in Section 2.2 of the initially submitted version. To co-evolve the policy and evaluator, a natural approach is self-play, as in GAN training. But GANs have real/fake labels, while current LLM responses lack quality rankings. EvoRubrics' discrimination reward separates responses but does not tell the evaluator which response is better. We overcome this with an extra anchor objective that teaches the evaluator which responses should score higher. This lets the policy and evaluator improve together.

W3/Q3.

(1) We count ties as errors and keep all samples in the accuracy denominator. In Tables 1 and 6, resolving a tie increases accuracy only when the preferred response ranks first without a tie. A tied ranking can already score well on nDCG, so correctly resolving it can improve accuracy more than nDCG.

(2) Correctly resolving ties improves ranking quality. Our method targets a key problem in post training. As policies improve, evaluators can give the same score to better and worse responses. We derive a solution through theoretical analysis (Section 3). Our experiments show that it works. DynamicRubric distinguishes previously tied responses and scores the better one higher (Section 4).

W4/Q4/Q6.

(1) Reward reliability. Our anchor design guards against reward hacking by keeping ranked anchors hidden from the generator (Section 3.4). The generated rubrics are evaluated on these unseen responses and rewarded for ranking better responses higher. We also shuffle the response list and average repeated verifier scores to stabilize rewards.

(2) Robustness. Our theory establishes how discriminability and correct ranking support policy improvement (Section 3). We observe improved policy performance. Our robustness checks show consistent gains across multiple verifiers, judges and benchmarks. We also manually inspect the responses and rubrics in our case study (Appendix H). Online A/B tests show improvements in a real industrial deployment serving tens of millions of user requests daily (Section 4.5).

(3) Co-evolution. In production, user preferences change over time. We refresh anchors to keep DynamicRubric aligned with these changing preferences and support continued improvement (Section 4.5).

Q1.

Yes. DynamicRubric also improves performance with a much larger production model. We obtain statistically significant gains in our industrial deployment serving tens of millions of user requests daily. Section 4.5 of the initially submitted version states:

These gains demonstrate that DynamicRubric scales to a much larger production model.

Q5.

Our training data exclude all prompts with potential overlap with the evaluation benchmarks, including UltraFeedback. We also hold out the entire LMSYS subset for evaluation (Appendix C.1).

We appreciate your recognition of our contribution. We hope these clarifications address your concerns and strengthen your support for our work.

Thank you for the clarifications, and particularly the explanation of tie handling and the additional details on data overlap.

My main concern remains the lack of controlled comparisons that isolate the effects of response-set conditioning and generator training. Regarding scaling, the larger production model is encouraging, but without quantitative results or model details, it is difficult to assess how well the method scales with policy size. I also think the theoretical justification remains limited.

Overall, the empirical results are promising, but these concerns have not been fully addressed. I will maintain my score.

Thank you for the follow-up.

(1) Training improves the rubric generator’s ranking accuracy. On AEOLLM, Qwen3-8B’s accuracy rises from 17.8 to 51.2 (Table 1). Both versions see the same candidate answers and use the same verifier, so this gain comes from training the generator.

(2) The policy performs better with our 8B generator that sees the candidate answers than with RaR’s 235B generator that sees only the prompt. Both methods train a Qwen3-8B policy. DynamicRubric scores higher on all four generation benchmarks (Table 2).

We hope these comparisons address your main concern. Thank you for taking another look at our work.

Reviewer_h6kkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies a problem in RL from evaluator feedback. The authors argue that as the LLM policy improves, the evaluators also need to evolve to prevent sampled candidate responses from converging in quality, or collapsing score gaps between responses. They introduce DynamicRubric, a framework that alternates between training a rubric generator conditioned on the models' response sets and optimizing the policy. Experiments are conducted using Qwen-8B and Llama-8B models, and the authors show that this approach outperforms supervision from larger pretrained reward models.

Critical strengths
  • The paper studies an important problem in self-improving LLMs.
  • The authors conduct extensive experiments with strong empirical results and some nice analyses, such as generalization to different backbones and verifiable tasks.
Critical weaknesses
  • My main concern is the novelty of the paper, as the problem and the approach do not seem to be new. The problems of evolving/dynamic rubrics or co-evolving policies and rubrics are already studied in previous work [1, 2]. Moreover, the approach looks very similar to EvoRubrics [2], which also uses bilevel RL with GRPO and includes a reward for score discrimination.
  • Some claims and theoretical derivations in the paper, especially in Sec 3.1-3.2 need further clarification. For example, in lines 171–172, the authors claim that "these gaps collapse, the optimization signal exposed by the evaluator shrinks, and policy optimization becomes less effective on the sampled response set". This is not well justified by the theoretical results or the experiments. In fact, the authors use GRPO for optimization, which standardizes rewards for responses within each group. Moreover, the derivation in Sec 3.2 relies on the assumption that E(x,y)∈RE(x, y) \in R. However, the rubrics appear to be discrete and binary.

[1] Zhang, Junkai, et al. "Chasing the tail: Effective rubric-based reward modeling for large language model post-training." International Conference on Learning Representations 2026.

[2] Ding, Hongxin, et al. "EvoRubrics: Dynamic Rubrics as Rewards via Adversarial Co-Evolution for LLM Reinforcement Learning." arXiv preprint arXiv:2606.23038 (2026).

Questions for authors
  • Why do the authors not include any other baselines in Figure 2?
  • For deployment, what are the actual details regarding the settings (models) and results?
3 comments

We thank Reviewer h6kk for the valuable feedback. Here are our responses.

W1.

Actually, we have already compared related work on co-evolution in Section 2.2 of the initially submitted version. Our work advances both the theory and practice of LLM co-evolution. We highlight the novelty of our work below.

(1) We theoretically formulate how LLM policies and evaluators should co-evolve during post training (Section 3).

(2) To co-evolve the policy and evaluator, a natural approach is self-play, as in GAN training. But GANs have real/fake labels, while current LLM responses lack quality rankings. EvoRubrics' discrimination reward separates responses but does not tell the evaluator which response is better. We overcome this with an extra anchor objective that teaches the evaluator which responses should score higher. This lets the policy and evaluator improve together.

(3) Our gains transfer from open domain generation to math, coding and scientific QA (Figure 3).

(4) Our model serves tens of millions of user requests daily in a fully deployed industrial AI answering system. Online A/B tests show statistically significant gains in total search volume, user duration and positive user behaviors (Section 4.5).

MethodCo-evolving rubric generatorGeneral domainTheoretical analysis of co-evolutionPerformance transfer to verifiable tasksIndustrial deployment
Chasing the TailNoYesNoNoNo
EvoRubricsYeshealth domainNoNoNo
DynamicRubricYesYesYesYesYes

W2. Theory

(1) Our work studies evaluator failures as policies improve during post training. The evaluator can score worse responses higher than better ones. GRPO cannot correct these errors through score normalization. We study this phenomenon in common post training paradigms and use GRPO only as the specific RL algorithm in our implementation.

(2) The aggregated evaluator score is a real value and satisfies the theoretical assumption. Eq. 12 combines binary rubric judgments through a weighted average, giving E(x,y∣C)∈[0,1]⊂RE(x,y\mid C)\in[0,1]\subset\mathbb R.

Q1.

Figure 2 tests the continuous co-evolution of DynamicRubric, so it does not include external baselines. We compare our co-evolution with continued training under old supervision. Both the evaluator and policy improve more with co-evolution.

Other baseline results are reported in detail in Tables 1 and 2. For example, on Personalized-RB, our evaluator (73.3) outperforms Skywork-27B (64.7) and Nemotron-70B (68.6). On AlpacaEval2, our policy (67.1) outperforms policies trained with Nemotron-70B (35.2) and RaR (58.5).

We also test whether training the other baselines for longer helps. Their performance stays flat or drops. This is the evaluator failure we address in W2. DynamicRubric fixes this by co-evolving the policy and evaluator.

Q2.

We withhold model and deployment details for anonymity and will disclose them in the camera-ready version. We have noted the confidentiality of model details in the Ethics Statement section.

Our industrial AI answering system serves tens of millions of user requests daily. A/B tests show statistically significant gains in total search volume, user duration and the number of positive user behaviors (Section 4.5). Our model fully replaces the previous model and handles all online traffic, showing its performance and industrial value.

We hope these responses address your concerns and would appreciate your reconsideration of our work.

Thank you for your response. It clarified some of my concerns. However, I am still unconvinced about the paper's novelty.

The discussion of the domain, performance transfer to verifiable tasks, and deployment is helpful. However, I do not find these arguments sufficient to justify the novelty, especially in the current form of the paper. The paper should strengthen the motivation around the problem and method, along with performance comparisons with prior work.

Regarding EvoRubrics, thank you for the clarification. However, I think this is not entirely accurate. While EvoRubrics' discrimination reward does not determine which response is better, the alignment and constructiveness rewards already provide quality guidance through reference rubrics. Therefore, the proposed anchor objective does not seem fundamentally different in motivation.

We discussed EvoRubrics in Section 2.2 of our September submission, about three months after its June preprint appeared. Thank you for recognizing the value of our transfer and deployment results.

Our motivation comes from a common problem in LLM post-training. As the policy improves, a fixed evaluator can give the same score to answers of different quality. The policy then earns no extra reward by producing the better answer more often. We therefore adapt the evaluator to the policy’s current answers and use its scores for the next policy update. Eq. 18 gives a lower bound on the expected score advantage of better answers in terms of our discriminability and anchor objectives. Our theory identifies this score advantage as the learning signal an evaluator must provide to guide policy improvement during post-training.

Beyond this theoretical result, consider how EvoRubrics and DynamicRubric handle the same ranking error. Suppose A is preferred to B. A rubric can match the reference and help improve both drafts yet still rank B first. Its alignment and constructiveness rewards in EvoRubrics can both be high. On an anchor pair hidden from our generator, ranking B first lowers the anchor reward. Widening B’s lead lowers it further. This discourages reward hacking through score gaps that favor the worse answer.

We hope this clarifies our motivation for co-evolution and for using ranked anchors.

Reviewer_LVVrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether evolving the rubrics together with the policy model can improve RL training. The idea is to generate rubrics based on the sampled responses from policy model itself using a rubrics generator, instead of using a fixed rubrics criteria. To evolve the rubrics, the generator is trained to distinguish responses from policy model and also preserve the scoring correctness using a set of fixed anchor data. The policy model is then trained with these rubric scores, and the method iteratively refines both the policy and rubric generator. Experiments show that DynamicRubric helps improving the performance on open-domain generation benchmarks and also provide benefits to verified tasks, such as math.

Critical strengths
  1. The investigated problem is critical. Training model with co-evolved, dynamic rubrics help continuously improve model behavior.
  2. The proposed method is well-motivated and effective.
  3. The proposed DynamicRubric is effective at improving the performance on open-domain generation benchmarks.
Critical weaknesses
  1. The proposed method may hard to scale. For tasks with long-context generation or agentic task, it may exceed the context window.
  2. While training the rubrics generator, the reward does not check the ranking correctness of policy responses and it favors rubrics that split the responses into two equally sized groups. It is unclear whether this supervision works when responses remain similar within a sampled group.
  3. The distinction reward does not consider the following scenario: if different rubric items are complementary, for example, item A gives 1 when item B gives 0 and vice versa. Even if they all split answers effectively, their summed score can not distinguish the same answer.
Questions for authors
  1. How to adapt DynamicRubric to verified tasks. How to effectively collect and annotate the ranking among different solutions, where their answers are correct but differ in solution pattern.
  2. How to deal with the scenario when answers exhibit similar pattern. In this case, what is the effect of distinction reward?
1 comment

We thank Reviewer LVVr for the valuable feedback. Here are our responses.

W1.

We split long response sets to fit the context window. Different response sets can produce different rubric sets. So we repeat rubric generation in two ways: using different splits and shuffling the response order. Then we combine the resulting rubric sets to improve robustness. This is like repeatedly tossing a needle in a Monte Carlo experiment to approximate π, and we observe that it stabilizes training. We have included this implementation in our initially submitted code.

W2.

Our joint objective teaches the generator to distinguish response quality.

(1) We train the rubric generator with both discriminability and anchor ranking supervision: Jgenerator=Jdisc+JanchorJ_{\mathrm{generator}} = J_{\mathrm{disc}} + J_{\mathrm{anchor}}. Both objectives evaluate the same generated rubric set:

ObjectiveEvaluated responsesTraining signalChecks ranking correctness?
DiscriminabilityCurrent policy responsesSeparate responsesNo
AnchorRanked anchors for the same promptScore better responses higherYes

(2) Our theoretical analysis guarantees that this joint objective works. Section 3.4 shows this combination forms a lower bound on the expected score gap. Section 3.1 shows that this gap is the learning signal for policy improvement. So, optimizing both objectives raises this lower bound.

(3) Our ablation shows that the two objectives work together. On AEOLLM, accuracy drops from 51.2 to 47.3 without the anchor objective and to 38.8 without the discriminability objective (Table 3).

Q2.

Our discriminability objective encourages more specific rubrics when responses follow similar patterns. A broad rubric that all responses satisfy earns zero discriminability reward. The generator is rewarded for finding differences between these responses, while the anchor objective guides which response should score higher. We have included a case study in Appendix H to show this.

W3.

As explained in W2, apart from the discriminability objective, our anchor objective checks the ranking of the final combined scores. So, the generator is rewarded for breaking the tie and giving the better response a higher score. Our method therefore still works in the case you describe.

Q1.

DynamicRubric can directly supervise thinking patterns. Given reasoning trajectories and their preference rankings, our generator learns to generate rubrics that evaluate the reasoning process itself. This provides process supervision for verifiable tasks.

As for the source of preference labels, our industrial deployment handles tens of millions of user requests daily. This lets us directly use users' preferences over thinking patterns as anchor supervision. Specifically, we use A/B tests to check which patterns users prefer through changes in total search volume, user duration and the number of positive user behaviors (Section 4.5).

Authors' comment on W2/W3/Q2:

Actually, these are exactly the same issues we face in developing our method. The discriminability objective alone does not tell the generator which response is better. Our setup is similar to GANs, but their training uses real/fake labels for the discriminator. However, LLM-generated answers lack ground truth quality rankings, which is a central challenge in LLM self-evolution. We start from our theoretical analysis and overcome this with two objectives, one for discriminability and the other for the correct optimization direction. This joint objective is a key strength of our work.

We hope these responses address your concerns and would appreciate your reconsideration of our work.

Comments on the paper

2 comments

Dear Reviewers,

Kind reminder that the deadline of the discussion period is Nov. 18 so please please read the reviewers' response and take a minute to write an acknowledgment and confirm your final evaluation, as soon as possible.

Thanks a lot for your help.

AuthorsOpenReview ↗

General Response

We thank the reviewers for the discussion.

We address a common problem in LLM post-training. As the policy improves, a fixed evaluator can give a better answer the same score as a worse one. Our theoretical analysis leads to DynamicRubric, a method for co-evolving the policy and its evaluator. A model trained this way now handles tens of millions of user requests a day in our production AI question answering system.

We are glad the reviewers found the problem important and our approach well motivated. They also highlighted the experimental evidence for co-evolution and found the production results encouraging. Here we summarize our discussion.

Reviewer LVVr

  • What if rubric items favor different answers and the total scores tie?

    • We train the rubric generator with two objectives. One favors items that tell answers apart. The other, our anchor objective, penalizes a tie in the final scores when we know which answer is better. Removing either objective lowers ranking accuracy in our ablation.
  • Does training stay stable when long response sets are split?

    • We test different groupings and answer orders. Training stays stable when we combine the resulting rubrics.
  • How can DynamicRubric extend to verifiable tasks when different reasoning paths reach the same correct answer?

    • We can use preferences from our online users to rank reasoning trajectories, which then serve as anchors.

Reviewer h6kk

  • How does DynamicRubric differ from other work on co-evolution, such as EvoRubrics?

    • We give a theory of how the policy and evaluator should co-evolve during post-training. We show why the evaluator needs to adapt to the policy’s current answers and derive its training objectives from that analysis.
  • Does updating the evaluator improve the policy in practice?

    • The same starting policy improves more with the updated evaluator than with the old one. The gains transfer to verifiable tasks. Our model performs better than its predecessor in online A/B tests and can continue to improve as we refresh its anchors in production.

Reviewer jmmZ

  • Does the policy do better when its rubric generator sees the answers?

    • Our 8B generator sees the answers. RaR’s 235B generator sees only the prompt. With the same backbone, the policy trained with our rubrics performs better.
  • Do the gains hold with further training and on a larger model?

    • We see gains in the second round and on a much larger production model. We refresh the production evaluator’s anchors as user preferences change.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.