- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes DynamicRubric, a framework that co-evolves LLM evaluators and policies using response-set-conditioned rubrics. A rubric generator produces weighted binary evaluation criteria and is optimized using discriminability and anchor-ranking rewards. The generator and policy are alternately updated with GRPO. Experiments demonstrate competitive evaluator performance against larger models and improvements in open-ended generation, verifiable tasks, and production deployment.
Critical strengths
- Well-motivated approach: Dynamically adapting evaluation criteria to current-policy responses is an intuitive and promising approach to improving reward supervision.
- Strong empirical results: The trained 8B rubric generator achieves competitive performance against substantially larger reward models, while the resulting policy outperforms strong baselines across multiple benchmarks.
- Evidence for co-evolution: Controlled experiments demonstrate that updating evaluators using current-policy responses provides better supervision than using stale evaluators.
- Broad evaluation: The improvements are validated across multiple benchmarks, model backbones, verifiers, and LLM judges.
Critical weaknesses
- Limited theoretical justification: The theoretical results largely follow from standard properties of linear objectives and function-class inclusion. Moreover, the emphasis on raw score gaps is not fully reconciled with GRPO's reward normalization, and the theoretical bound relies on an additional calibration assumption.
- Insufficient baseline comparisons: The experiments do not fully isolate the contributions of response-set conditioning and generator training. Comparisons with closely related co-evolution methods, such as EvoRubric and EvoRubrics, are also missing.
- Limited analysis of discriminability: The reported accuracy improvements may partly reflect reduced score ties rather than improved ranking quality. This concern is particularly relevant because listwise nDCG improvements are relatively modest.
- Limited analysis of robustness and stability: Potential reward hacking, response-length biases, and long-term co-evolution stability are not sufficiently investigated. The co-evolution experiments cover only two rounds.
Questions for authors
- Scaling: Have the authors explored training larger rubric generators (e.g., 32B)? Does DynamicRubric continue to improve performance as model size increases?
- Baseline comparisons: Can the authors compare against a trained prompt-only rubric generator under matched conditions and explain the differences from EvoRubric and EvoRubrics?
- Score ties: How much of the evaluator accuracy improvement comes from reducing tied scores? Can the authors report tie rates and clarify the tie-handling protocols in Tables 1 and 6?
- Reward reliability: Can the authors provide further evidence connecting evaluator discriminability to downstream policy improvements and investigate potential reward hacking or response-length biases?
- Data contamination: Have the authors checked for overlap between the Nectar training data and the evaluation benchmarks, particularly UltraFeedback?
- Long-term stability: How does performance evolve beyond two co-evolution rounds? Does it continue to improve or eventually saturate?
3 comments
Thank you for the clarifications, and particularly the explanation of tie handling and the additional details on data overlap.
My main concern remains the lack of controlled comparisons that isolate the effects of response-set conditioning and generator training. Regarding scaling, the larger production model is encouraging, but without quantitative results or model details, it is difficult to assess how well the method scales with policy size. I also think the theoretical justification remains limited.
Overall, the empirical results are promising, but these concerns have not been fully addressed. I will maintain my score.
Thank you for the follow-up.
(1) Training improves the rubric generator’s ranking accuracy. On AEOLLM, Qwen3-8B’s accuracy rises from 17.8 to 51.2 (Table 1). Both versions see the same candidate answers and use the same verifier, so this gain comes from training the generator.
(2) The policy performs better with our 8B generator that sees the candidate answers than with RaR’s 235B generator that sees only the prompt. Both methods train a Qwen3-8B policy. DynamicRubric scores higher on all four generation benchmarks (Table 2).
We hope these comparisons address your main concern. Thank you for taking another look at our work.
We thank Reviewer jmmZ for the valuable feedback. Here are our responses.
W1. Theory
(1) Our theory explains how LLM policies and evaluators should co-evolve during post training. Other co-evolution methods lack this theoretical analysis. We explain why evaluators fail as policies improve and derive a solution (Section 3). Our experiments show that this solution works (Section 4).
(2) Our work studies evaluator failures as policies improve during post training. The evaluator can score worse responses higher than better ones. GRPO cannot correct these errors through score normalization. We study this phenomenon in common post training paradigms and use GRPO only as the specific RL algorithm in our implementation.
W2/Q2. Comparisons
(1) We have already isolated the effects of response access and continued training in the initially submitted version.
Response access. Our DR-Generator-8B (51.2 on AEOLLM, for example) outperforms the strong Qwen3-32B baseline (29.7) on all six accuracy benchmarks. We also compare the strong Nemotron-70B baseline with and without responses (UltraFeedback 72.8 → 77.4, Personalized-RB 68.6 → 61.9) (Table 1). Simply seeing responses does not guarantee better evaluation.
Continued training. Figure 2(a) starts from the same generator and continues training on either old or current policy responses. Performance stays flat or drops with old responses, while seeing current responses brings further gains.
(2) We have already compared these related works in Section 2.2 of the initially submitted version. To co-evolve the policy and evaluator, a natural approach is self-play, as in GAN training. But GANs have real/fake labels, while current LLM responses lack quality rankings. EvoRubrics' discrimination reward separates responses but does not tell the evaluator which response is better. We overcome this with an extra anchor objective that teaches the evaluator which responses should score higher. This lets the policy and evaluator improve together.
W3/Q3.
(1) We count ties as errors and keep all samples in the accuracy denominator. In Tables 1 and 6, resolving a tie increases accuracy only when the preferred response ranks first without a tie. A tied ranking can already score well on nDCG, so correctly resolving it can improve accuracy more than nDCG.
(2) Correctly resolving ties improves ranking quality. Our method targets a key problem in post training. As policies improve, evaluators can give the same score to better and worse responses. We derive a solution through theoretical analysis (Section 3). Our experiments show that it works. DynamicRubric distinguishes previously tied responses and scores the better one higher (Section 4).
W4/Q4/Q6.
(1) Reward reliability. Our anchor design guards against reward hacking by keeping ranked anchors hidden from the generator (Section 3.4). The generated rubrics are evaluated on these unseen responses and rewarded for ranking better responses higher. We also shuffle the response list and average repeated verifier scores to stabilize rewards.
(2) Robustness. Our theory establishes how discriminability and correct ranking support policy improvement (Section 3). We observe improved policy performance. Our robustness checks show consistent gains across multiple verifiers, judges and benchmarks. We also manually inspect the responses and rubrics in our case study (Appendix H). Online A/B tests show improvements in a real industrial deployment serving tens of millions of user requests daily (Section 4.5).
(3) Co-evolution. In production, user preferences change over time. We refresh anchors to keep DynamicRubric aligned with these changing preferences and support continued improvement (Section 4.5).
Q1.
Yes. DynamicRubric also improves performance with a much larger production model. We obtain statistically significant gains in our industrial deployment serving tens of millions of user requests daily. Section 4.5 of the initially submitted version states:
Q5.
Our training data exclude all prompts with potential overlap with the evaluation benchmarks, including UltraFeedback. We also hold out the entire LMSYS subset for evaluation (Appendix C.1).
We appreciate your recognition of our contribution. We hope these clarifications address your concerns and strengthen your support for our work.