- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes EvoRS, a self-evolving reward system for reinforcement learning on open-ended tasks. It represents the reward system as an executable Reward-DAG and uses on-policy rollouts and reward traces to dynamically update reward criteria, scoring mechanisms, and signal composition. Experiments on writing, roleplay, and agentic tasks show improvements over existing baselines, with ablations supporting the effectiveness of the main components.
Critical strengths
- Well-motivated approach with practical value.
Fixed reward systems may suffer from reward hacking, coverage gaps, and reward saturation as the policy evolves. Beyond dynamically updating rubrics, EvoRS also adapts scoring mechanisms and reward composition, providing a more comprehensive framework for reward optimization in open-ended tasks.
- Solid experimental results, especially on roleplay tasks.
EvoRS achieves substantial improvements over existing baselines on CoSER, supported by three independent LLM judges. The paper also provides reward reliability analyses, experiments with different policies and reward models, and component ablations, offering good evidence for the effectiveness of the method.
Critical weaknesses
- The reliability of reward failure diagnosis is not sufficiently validated.
EvoRS uses on-policy responses and reward traces to decide whether to update the reward system. However, it is unclear whether the method can reliably distinguish reward failures from insufficient policy training or normal convergence. For example, lower reward variance does not necessarily indicate an unreliable reward. Although matched replay evaluates candidate updates, it mainly examines reward behavior rather than directly showing that updating the reward leads to better downstream policy quality than continuing training with the original reward.
- Stronger simple reward baselines are needed to establish the necessity of evolution.
EvoRS outperforms existing static and dynamic rubric methods, but it is unclear whether these tasks truly require complex agentic reward evolution. For example, some roleplay improvements may come from better penalties for repetition, verbosity, or low-quality content. Although the paper evaluates a fixed final Reward-DAG, it does not compare against well-designed multi-component static rewards or simple scheduled reward updates. This makes it difficult to isolate the benefit of on-policy evolution from improvements in reward design itself.
Questions for authors
Reward diagnosis: How does EvoRS distinguish reward failures from insufficient policy training or normal convergence? Is there evidence that updating the reward at selected evolution steps leads to better downstream policy quality than continuing with the original reward?
Baselines: Have the authors considered simpler baselines, such as static multi-component rewards with repetition or verbosity penalties, or scheduled reward updates? Such comparisons would help establish whether complex on-policy evolution is necessary.
2 comments
Weakness 2
- Stronger simple reward baselines are needed to establish the necessity of evolution.
EvoRS outperforms existing static and dynamic rubric methods, but it is unclear whether these tasks truly require complex agentic reward evolution. For example, some roleplay improvements may come from better penalties for repetition, verbosity, or low-quality content. Although the paper evaluates a fixed final Reward-DAG, it does not compare against well-designed multi-component static rewards or simple scheduled reward updates. This makes it difficult to isolate the benefit of on-policy evolution from improvements in reward design itself.
Baselines: Have the authors considered simpler baselines, such as static multi-component rewards with repetition or verbosity penalties, or scheduled reward updates? Such comparisons would help establish whether complex on-policy evolution is necessary.
We thank the reviewer for this suggestion. We added two ablations: a hand-designed static multi-component reward, and scheduled reward updates.
1. Hand-designed static multi-component reward. Following the characteristics of the writing and roleplay settings, we took the form of a continuous scorer combined with a multiplicative flaw penalty and used human priors to fix a fairly general multi-node system, then trained with that reward system held fixed throughout.
2. Scheduled reward updates. Again following human priors, we expanded an initial DAG step by step into a 6-node reward system, applying one predefined expansion every 5 steps during training:
| Variant | WB ↑ | CoSER ↑ |
|---|---|---|
| Qwen3-4B base | 54.894 | 60.909 |
| Only criteria-level adaptation | 56.593 | 61.155 |
| Fixed final Reward-DAG | 56.308 | 60.573 |
| Fixed multi-node RS (hand-designed) | 56.013 | 63.345 |
| Scheduled Reward-DAG Replay (scheduled updates) | 55.331 | 61.042 |
| Full EvoRS | 57.001 | 65.676 |
- A fixed multi-node reward system from human priors helps early on, but it cannot replace evolving with the policy distribution. Looking further at the scores of the intermediate checkpoints, the fixed reward beats EvoRS over the first 60 steps, but then saturates and stops improving.
- Scheduled updates bring little gain. The predefined update directions do not necessarily match the behavior the policy actually develops. In the writing setting, for example, the hack of padding responses with numbers never appeared, which left the reward signal off target.
- The final DAG, although it also came out of evolution, brings little gain. Looking at its component design, we find that its penalty terms target the hacking behavior of one specific policy late in evolution; introducing them early does not fit the policy and holds training back instead. This shows that the gain comes not only from better reward design, but also from whether each component is introduced at a time that matches the current policy distribution.
Response
We thank the reviewer for the careful reading and the suggestions.
We will revise the paper as follows:
Answers to the specific comments follow.
Weakness 1
We understand this concern. We read it as two questions: how we tell a reward failure apart from normal convergence, and whether updating the reward really improves downstream training.
On the first question, telling reward failure apart from normal convergence, we rely on the following mechanisms:
1. Reward statistics only serve as diagnostic signals; they never trigger an update on their own. We agree that low variance does not mean the reward is unreliable. As described in Appendix C.5 and D.5, reward statistics are treated as health signals rather than as separate optimization targets, and high variance that comes only from noisy penalties or from score collapse is not taken as evidence of a better reward. What actually supports an update is the problems the agentic designer sees directly in the rollouts through sampling and comparison, such as responses that score high but read poorly.
2. We also use external metrics, independent of the reward, to separate failure from convergence. We additionally track how well the reward system's preference ranking over responses agrees with the rankings from a strong LLM judge and from humans (Sec. 6.1, Appendix E.4):
On the second question, whether updating the reward really improves downstream training:
We took three consecutive checkpoints where the update was relatively large, fixed the reward system right after each one, and ran three separate training runs in the roleplay setting, to see whether each single update is beneficial on its own: