acceptodds
Under review as a conference paper at ICLR 2027

EvoRS: On-Policy Self-Evolution of Reward Systems for Open-Ended Reinforcement Learning

Abstract

Open-ended reinforcement learning often relies on rubric-based rewards for tasks without directly verifiable answers. Yet the policy and reward system form a dynamic feedback loop: as the policy optimizes the current reward, an initially useful reward system may become unreliable due to reward hacking or reduced response discriminability. The reward system should therefore evolve rather than remain fixed during training. Existing dynamic-rubric methods adapt evaluation criteria, but reward failures can also arise from scoring mechanisms or signal composition. We introduce EvoRS, a self-evolving RL framework that evolves the reward system from on-policy experience, representing it as an executable Reward-DAG. Specifically, an agentic designer updates this system from on-policy rollouts and reward traces to maintain train-time reliability. Across writing and roleplay, EvoRS achieves the best quality under all three judges, outperforming the policy by \(2.107\) and \(4.767\) points, respectively, while reducing reward hacking and coverage failures and preserving reward informativeness. Ablations confirm that a comprehensive fixed reward system cannot remain reliable in open-ended tasks and must evolve throughout training.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Rubric-Based Rewards, Reward Hacking, LLM Agents, Open-ended Task

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_3ueqOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes EvoRS, a self-evolving reward system for reinforcement learning on open-ended tasks. It represents the reward system as an executable Reward-DAG and uses on-policy rollouts and reward traces to dynamically update reward criteria, scoring mechanisms, and signal composition. Experiments on writing, roleplay, and agentic tasks show improvements over existing baselines, with ablations supporting the effectiveness of the main components.

Critical strengths
  1. Well-motivated approach with practical value.

Fixed reward systems may suffer from reward hacking, coverage gaps, and reward saturation as the policy evolves. Beyond dynamically updating rubrics, EvoRS also adapts scoring mechanisms and reward composition, providing a more comprehensive framework for reward optimization in open-ended tasks.

  1. Solid experimental results, especially on roleplay tasks.

EvoRS achieves substantial improvements over existing baselines on CoSER, supported by three independent LLM judges. The paper also provides reward reliability analyses, experiments with different policies and reward models, and component ablations, offering good evidence for the effectiveness of the method.

Critical weaknesses
  1. The reliability of reward failure diagnosis is not sufficiently validated.

EvoRS uses on-policy responses and reward traces to decide whether to update the reward system. However, it is unclear whether the method can reliably distinguish reward failures from insufficient policy training or normal convergence. For example, lower reward variance does not necessarily indicate an unreliable reward. Although matched replay evaluates candidate updates, it mainly examines reward behavior rather than directly showing that updating the reward leads to better downstream policy quality than continuing training with the original reward.

  1. Stronger simple reward baselines are needed to establish the necessity of evolution.

EvoRS outperforms existing static and dynamic rubric methods, but it is unclear whether these tasks truly require complex agentic reward evolution. For example, some roleplay improvements may come from better penalties for repetition, verbosity, or low-quality content. Although the paper evaluates a fixed final Reward-DAG, it does not compare against well-designed multi-component static rewards or simple scheduled reward updates. This makes it difficult to isolate the benefit of on-policy evolution from improvements in reward design itself.

Questions for authors

Reward diagnosis: How does EvoRS distinguish reward failures from insufficient policy training or normal convergence? Is there evidence that updating the reward at selected evolution steps leads to better downstream policy quality than continuing with the original reward?

Baselines: Have the authors considered simpler baselines, such as static multi-component rewards with repetition or verbosity penalties, or scheduled reward updates? Such comparisons would help establish whether complex on-policy evolution is necessary.

2 comments

Response

We thank the reviewer for the careful reading and the suggestions.

We will revise the paper as follows:

  1. State in the main text what role reward statistics play in diagnosis, and add a branching comparison that continues training with the original reward versus the updated one;
  2. Add two baselines — a hand-designed static multi-component reward and scheduled reward updates — and say more about the Fixed final Reward-DAG ablation already in the submitted version.

Answers to the specific comments follow.

Weakness 1

  1. The reliability of reward failure diagnosis is not sufficiently validated.

EvoRS uses on-policy responses and reward traces to decide whether to update the reward system. However, it is unclear whether the method can reliably distinguish reward failures from insufficient policy training or normal convergence. For example, lower reward variance does not necessarily indicate an unreliable reward. Although matched replay evaluates candidate updates, it mainly examines reward behavior rather than directly showing that updating the reward leads to better downstream policy quality than continuing training with the original reward.

Reward diagnosis: How does EvoRS distinguish reward failures from insufficient policy training or normal convergence? Is there evidence that updating the reward at selected evolution steps leads to better downstream policy quality than continuing with the original reward?

We understand this concern. We read it as two questions: how we tell a reward failure apart from normal convergence, and whether updating the reward really improves downstream training.

On the first question, telling reward failure apart from normal convergence, we rely on the following mechanisms:

1. Reward statistics only serve as diagnostic signals; they never trigger an update on their own. We agree that low variance does not mean the reward is unreliable. As described in Appendix C.5 and D.5, reward statistics are treated as health signals rather than as separate optimization targets, and high variance that comes only from noisy penalties or from score collapse is not taken as evidence of a better reward. What actually supports an update is the problems the agentic designer sees directly in the rollouts through sampling and comparison, such as responses that score high but read poorly.

2. We also use external metrics, independent of the reward, to separate failure from convergence. We additionally track how well the reward system's preference ranking over responses agrees with the rankings from a strong LLM judge and from humans (Sec. 6.1, Appendix E.4):

Reliability checkGroupsPairwise agreement
Three-model LLM judge panel4800.8408
Human vs. LLM consensus200.71
  • Under normal convergence the scores rise overall while this agreement does not fall, whereas a reward failure shows up as a drop in preference agreement.
  • On most checkpoints EvoRS keeps both better statistical metrics and higher preference agreement, which shows that the distinctions it draws still match real preferences.

On the second question, whether updating the reward really improves downstream training:

  1. We took three consecutive checkpoints where the update was relatively large, fixed the reward system right after each one, and ran three separate training runs in the roleplay setting, to see whether each single update is beneficial on its own:

    SettingSelected stepCoSER ↑HR ↓CFR ↓
    Fixed at c15563.1828.3%14.5%
    Fixed at c26063.0506.1%12.5%
    Fixed at c36564.2156.3%13.0%
    Continued evolution (Full EvoRS)/65.6765.25%10.0%
  • No single update reliably improves the policy; the improvement comes from iterating continuously. In the intermediate checkpoint scores, c1 is above c3 at step 80 (62.5 vs 62.3) but below c3 at step 120, and all three fixed points fall below full continued evolution. One good update therefore only helps locally, while the overall gain needs continued iteration.
  • c2 costs a little quality relative to c1, but it does fix the Hack Rate. The c2 edit is the first attempt at long-response hacking and adds a penalty mechanism; this lowers overall training variance somewhat, but it holds the hacking down. c3 then drops the saturated rubric terms that carry no variance, fixing the variance problem while easing the over-penalization of reward hacking that c2 introduced.

Weakness 2

  1. Stronger simple reward baselines are needed to establish the necessity of evolution.

EvoRS outperforms existing static and dynamic rubric methods, but it is unclear whether these tasks truly require complex agentic reward evolution. For example, some roleplay improvements may come from better penalties for repetition, verbosity, or low-quality content. Although the paper evaluates a fixed final Reward-DAG, it does not compare against well-designed multi-component static rewards or simple scheduled reward updates. This makes it difficult to isolate the benefit of on-policy evolution from improvements in reward design itself.

Baselines: Have the authors considered simpler baselines, such as static multi-component rewards with repetition or verbosity penalties, or scheduled reward updates? Such comparisons would help establish whether complex on-policy evolution is necessary.

We thank the reviewer for this suggestion. We added two ablations: a hand-designed static multi-component reward, and scheduled reward updates.

1. Hand-designed static multi-component reward. Following the characteristics of the writing and roleplay settings, we took the form of a continuous scorer combined with a multiplicative flaw penalty and used human priors to fix a fairly general multi-node system, then trained with that reward system held fixed throughout.

2. Scheduled reward updates. Again following human priors, we expanded an initial DAG step by step into a 6-node reward system, applying one predefined expansion every 5 steps during training:

VariantWB ↑CoSER ↑
Qwen3-4B base54.89460.909
Only criteria-level adaptation56.59361.155
Fixed final Reward-DAG56.30860.573
Fixed multi-node RS (hand-designed)56.01363.345
Scheduled Reward-DAG Replay (scheduled updates)55.33161.042
Full EvoRS57.00165.676
  • A fixed multi-node reward system from human priors helps early on, but it cannot replace evolving with the policy distribution. Looking further at the scores of the intermediate checkpoints, the fixed reward beats EvoRS over the first 60 steps, but then saturates and stops improving.
  • Scheduled updates bring little gain. The predefined update directions do not necessarily match the behavior the policy actually develops. In the writing setting, for example, the hack of padding responses with numbers never appeared, which left the reward signal off target.
  • The final DAG, although it also came out of evolution, brings little gain. Looking at its component design, we find that its penalty terms target the hacking behavior of one specific policy late in evolution; introducing them early does not fit the policy and holds training back instead. This shows that the gain comes not only from better reward design, but also from whether each component is introduced at a time that matches the current policy distribution.
Reviewer_faFwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduce EvoRS, a framework that uses evolving reward system using an executable Reward-DAG. Compared to the existing work that uses evolving rubric, EvoRS further make revisions on its reward system, i.e., including rubric, scoring mechanism, and reward aggregation. Testing EvoRS on writing and roleplay tasks, it scored the best performance compared to other baselines.

Critical strengths

The high level motivation is interesting. Making evolution on the reward system itself could address a practical problem of rubric-based reward system in RL.

Critical weaknesses

The effectiveness of the proposed method is not validated enough, and there are many experiments needed to fully support authors claim.

  1. Need more fair baselines. I guess EvoRS uses more frequent GRM calls as the graph grows. The authors may use more LLM samplings of other baselines to match the computational cost with EvoRS. In addition, I believe EvoRubric [1]is more close baseline so I recommend adding it as a new baseline.
  2. Need more ablations. Specifically, a fixed multi-node reward system (e.g., a continuous scorer combined with a multiplicative flaw-penalty guard) used from the first step, and a variant that selects candidates with the variance-based rule of DR Tulu [2] instead of the designer’s judgment, would show whether the gains come from evolving the scoring mechanism and from the LLM designer, respectively.

[1] EvoRubric: Self-Evolving Rubric-Driven RL for Open-Ended Generation https://arxiv.org/abs/2605.29847 [2] DR Tulu: Reinforcement Learning with Evolving Rubrics for Deep Research https://arxiv.org/abs/2511.19399

Questions for authors

[Q1] Could you refine the figure 4 to enhance readability and conciseness? Figure 4 looks cute and beautiful, but it is quite hard to grab the essence of the framework. More concise one (e.g., using fewer texts and components) would work better.

[Q2] I am not convinced if the process for evaluating “a better reward system” (i.e., when selecting $D_{t+1}) works as intended. They can make errors when evaluating them, which can induce a significant error in training.

[Q3] Please add up/down arrows in Table 2’s header to denote which metric means higher the better or lower the better.

[Q4] Could the authors explain why the improvements on WritingBench (Table 3) seems marginal?

8 comments

Response

We thank the reviewer for the constructive comments, which help us organize and strengthen the experiments and arguments already in the paper.

We have made the following revisions and additions in response:

  1. Added EvoRubric as a baseline, plus ablations on a fixed multi-node reward system and on DR-Tulu-style variance-based selection.
  2. Expanded the discussion of reward-system updates and of the WritingBench results, with supporting experiments.
  3. Improved the tables in the paper, and planned a redesign of Figure 4.

Answers to the specific comments follow.

Weakness 1

Need more fair baselines. I guess EvoRS uses more frequent GRM calls as the graph grows. The authors may use more LLM samplings of other baselines to match the computational cost with EvoRS. In addition, I believe EvoRubric [1] is more close baseline so I recommend adding it as a new baseline.

We thank the reviewer for this suggestion. EvoRubric is indeed relevant work on open-ended settings; we now cite it and add it as a baseline. We have also counted the GRM calls each method actually makes, so that the GRM cost of the different baselines can be compared directly.

We first want to clarify that EvoRS controls cost by bounding the graph size (Appendix C.2). In a Reward-DAG, each complete execution path holds at most 3 GRM nodes; branches are allowed, but the graph holds at most 10 nodes in total, and program-level legality checks that the graph structure stays within this bound. This is a conservative setting, and harder settings could relax it, but every experiment in the paper uses it.

We then added the EvoRubric experiment and counted the GRM calls under all baselines for comparison.

  1. EvoRubric baseline. We keep EvoRubric's setting of sampling 4 rubrics and match everything else to EvoRS, using our local Qwen3.5-27B as the GRM in place of the gpt-oss-120b grader of the EvoRubric paper, and training Qwen3-4B:

    MethodWB ↑Writing HR ↓Writing CFR ↓CoSER ↑Roleplay HR ↓Roleplay CFR ↓
    Qwen3-4B base54.8947.7%27.8%60.9096.06%13.0%
    RaR56.54714.7%29.5%60.0008.52%22.0%
    EvoRubric56.58810.35%26.1%62.7468.97%13.5%
    EvoRS57.0016.5%19.9%65.6765.25%10.0%
  • EvoRubric beats the static-rubric baselines on both final quality and reward validity, but stays below EvoRS. This matches our main claim: evolving the criteria alone does not keep the reward valid, and the scoring mechanisms and the signal composition have to move with the policy distribution as well.
  1. GRM calls across baselines:

    MethodGRM calls ↓GRM calls per response ↓
    RLAIF≈ 11,5201.00
    RaR≈ 11,5201.00
    OpenRS≈ 16,3501.42
    EvoRS≈ 21,6601.88
    RLER (DR-Tulu)≈ 23,0402.00
    EvoRubric≈ 57,6005.00
  • In our setting, EvoRS stays within a reasonable range at 1.88 GRM calls per response, below the 2 calls of RLER (DR-Tulu) and the 5 calls of EvoRubric.

As stated in Appendix C.1, all baselines share the same rollout budget, batching and other settings. Since EvoRS already makes about as many GRM calls as the existing baselines, we do not raise their sampling cost here.

Weakness 2

Need more ablations. Specifically, a fixed multi-node reward system (e.g., a continuous scorer combined with a multiplicative flaw-penalty guard) used from the first step, and a variant that selects candidates with the variance-based rule of DR Tulu [2] instead of the designer's judgment, would show whether the gains come from evolving the scoring mechanism and from the LLM designer, respectively.

We agree these ablations matter, since they target the two key assumptions behind our method, and we have run both of the experiments the reviewer asks for.

We note that the paper already includes a variant that fixes the final evolved result (a multi-node reward system). That system, however, is what EvoRS's own evolution selected, and it may carry penalties aimed at hack patterns from one specific point in training, which is why it trains only moderately well.

  1. We therefore used human priors to fix a fairly general initial multi-node system — the continuous scorer combined with a multiplicative flaw-penalty guard that the reviewer suggests — and trained with it.
  2. We also updated the Reward-DAG with the variance-based rule of DR Tulu [2] (the RLER baseline in the paper) instead of the designer's judgment from comparative evidence, leaving the rest of the pipeline unchanged, so as to isolate the agentic designer on its own.
VariantWB ↑CoSER ↑
Qwen3-4B base54.89460.909
Fixed final Reward-DAG56.30860.573
Only criteria-level adaptation56.59361.155
Fixed multi-node reward system (hand-designed)56.01363.345
DR-Tulu-style variance-based evolution56.28662.892
Full EvoRS57.00165.676

a) A fixed multi-node reward system from human priors helps early on, but it cannot replace evolving with the policy distribution. Looking at the scores of the intermediate checkpoints, the fixed reward beats EvoRS over the first 60 steps, but then saturates and stops improving.

b) Replacing the designer's selection with the variance rule performs the same as doing no selection at all. Table 6 of the paper reports the w/o candidate selection ablation, which removes the selection mechanism, and its score is close to variance-based selection (62.892 vs 62.494). In case-level diagnosis we find that the designer, working from comparative evidence and similar signals (see Appendix E.2), locates more reward hacking and coverage problems than a statistical rule such as variance can.

Question 1

[Q2] I am not convinced if the process for evaluating "a better reward system" (i.e., when selecting Dt+1D_{t+1}) works as intended. They can make errors when evaluating them, which can induce a significant error in training.

We understand this concern. EvoRS limits how much any single update can change, and relies on continued evolution for its gains.

  1. legality checks and update rejection keep failed updates out to a reasonable extent (Appendix C.2), and we also cap how much each update may change, so that no single edit reshapes the reward system enough to disturb training.

  2. we agree these mechanisms cannot remove evolution bias entirely. In our method, an update is not required to improve the overall training result every time. What continued evolution gives is the overall trend: the reward system follows the shift in the policy distribution and catches problems as they appear, which is what improves the final result.

Concretely, we took three consecutive checkpoints where the update was relatively large, fixed the reward system right after each one, and ran three separate training runs in the roleplay setting, to see whether each single update is beneficial on its own:

SettingSelected stepCoSER ↑HR ↓CFR ↓
Fixed at c15563.1828.3%14.5%
Fixed at c26063.0506.1%12.5%
Fixed at c36564.2156.3%13%
Continued evolution (Full EvoRS)/65.6765.25%10.0%
  • No single update reliably improves the policy; the improvement comes from updating continuously. In the intermediate checkpoint scores, c1 is above c3 at step 80 (62.5 vs 62.3) but below c3 at step 120, so one good update only helps locally, while the overall gain needs continued evolution.
  • c2 costs a little quality relative to c1, but it does fix the Hack Rate. The c2 edit is the first attempt at long-response hacking and adds a penalty mechanism; this lowers overall training variance somewhat, but it holds the hacking down. c3 then drops the saturated rubric terms that carry no variance, fixing the variance problem while easing the over-penalization of reward hacking that c2 introduced.

We think this is a valuable question: defining "a better reward system" is hard, because coverage, hack resistance and difficulty can trade off against each other. What our method aims for is a reward system that helps training as much as it can at each step — not a guarantee that every step is better than the last, but a guarantee that every step keeps diagnosing the shift in the policy distribution and adapting to it.

Reviewer_6Lg8OpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces an adaptive agentic reward designer for rubric-based rewards in open-ended RL for LLMs. It claims that static rubric-based rewards are susceptible to saturation and reward hacking, reducing response discriminability. Moreover, it argues that dynamic rubric methods remain underperforming because they change only the rubrics, not the scoring mechanism. To address this problem, the paper proposes a self-evolving reward system based on on-policy experience, in which an agentic reward designer adapts both the rubric and the scoring mechanism based on the latest policy score and overall on-policy experience. The reward system is represented as a DAG, which the agentic system adapts according to the policy's experience and rewards. The paper shows the effectiveness of their method in roleplay and Writing benchmarks.

Critical strengths
  • The motivating analysis section makes the paper more convincing and makes it clear where the method might be useful.
  • The method shows consistent improvement in quality, reliability, and signal metrics in both benchmakrs compared to the baselines.
  • The analysis shows each component of the method contributes to its performance.
Critical weaknesses
  • Using an agentic system to adaptively change the reward makes the method more expensive, and depending on the quality of the LLM, the method might be expensive to scale.
  • The results are only shown on Qwen models.
Questions for authors
  • Is it possible to show some results on other open-weight LLMs of the same size as Qwen? Like Gemma or some Mistral models?
2 comments

Response

We thank the reviewer for the helpful comments, which help us improve the completeness and clarity of the paper.

We have made the following revisions:

  1. We now report the extra cost of our method more clearly in the appendix, together with a cost-quality comparison across agentic designer configurations.
  2. We added training runs on other policies, including Gemma4, to show that the method still helps there.

Detailed answers to the reviewer's comments follow.

Weakness 1

Using an agentic system to adaptively change the reward makes the method more expensive, and depending on the quality of the LLM, the method might be expensive to scale.

We understand this concern: introducing an agentic system does add cost. Appendix C.1 / Table 10 reports the extra training time our method takes:

MethodAvg. step time ↓Avg. agent update timeAgent interval
RLAIF511.3s––
RaR663.8s––
RLER718.4s––
EvoRS, training only753.0s––
EvoRS, with designer801.5s242.8s5 steps
OpenRS1037.4s––
  • In training time, the agentic designer adds only about 6.0%. This is still well below the dynamic-rubric method OpenRS, so we believe the cost is acceptable.

Appendix C.1 / Table 11 also reports how different agentic designers perform in our framework, including DeepSeek-V4-Flash as a cheap option. Beyond the submitted version, we have now also measured how many tokens the four configurations consume:

Designer configurationLocal diagnostic/edit behaviorCoSER Overall ↑Avg. tokens per training run ↓
Codex SDK + GPT-5.4Completed workflow65.676≈ 16.35M
OpenCode + DeepSeek-V4-FlashCompleted workflow65.93≈ 24.71M
KimiCode + Kimi-K2.5Completed workflow64.34≈ 22.18M
OpenCode + MiniMax-M2.5Frequent tool/edit failures61.44/
  • In tokens, the agentic designer uses only 15–25M in total. The designer audits one full batch every N steps, so it is called few times, and that count does not depend on the number of rollouts or the batch size. Scaling training up therefore does not add designer cost.
  • A cheap model can act as the agentic designer and still train well, as long as it uses the diagnostic tools we provide correctly. DeepSeek-V4-Flash reaches comparable quality here for only $5–8 of extra cost. What the designer model needs is reliable tool use: MiniMax-M2.5, for example, reaches only 61.44 because its tool and edit calls often fail

Weakness 2

The results are only shown on Qwen models. Is it possible to show some results on other open-weight LLMs of the same size as Qwen? Like Gemma or some Mistral models?

We thank the reviewer for this suggestion. We added EvoRS training on Gemma4 in the roleplay setting, to check whether the gains depend on the Qwen model family.

PolicyMethodCoSER ↑HR ↓CFR ↓
Qwen3-4BBase60.9096.06%13.0%
RaR60.0008.52%22.0%
EvoRS65.6765.25%10.0%
Gemma4-E2BBase74.5213.15%17.14%
RaR76.4413.85%16.00%
EvoRS77.6911.87%13.50%
  • Gemma4-E2B also benefits from EvoRS, and the gain is larger than RaR's. Unlike on the Qwen policies, RaR is hacked much less on Gemma4, but it still does not match EvoRS.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.