- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Rubric-based RL compresses rubric evaluations into a scalar reward, discarding actionable feedback, while fixed rubrics can lead to suboptimal policies. The paper uses an external feedback model to extract natural-language experiences from rubrics and responses, improves the policy via on-policy context distillation, then repurposes the policy as a rubric generator for further rubric training, enabling rubric–policy co-evolution.
Critical strengths
- Rubrics are no longer fixed but evolve alongside policy training, while the capability of the rubric generator is also continuously improved.
- Rubric-based natural-language experiences provide more specific guidance than scalar rewards.
- The training pipeline is relatively complex, and demonstrating stable training would itself require substantial effort.
Critical weaknesses
- An ablation with a fixed rubric generation policy is missing to demonstrate the effectiveness of rubric training.
- What strategies are used to select high-quality rubrics, and why are these strategies reliable indicators of rubric quality?
- Why does rubric training use the highest-scoring and lowest-scoring responses as teacher context, rather than directly applying SFT to learn rubrics deemed high-quality by the authors? Moreover, since these scores are determined by the current rubrics, could using these responses as context introduce self-bias?
- How do the rubrics and experiences evolve throughout training? I suggest highlighting the key changes identified by the authors in the case studies, rather than presenting entire passages. Additional comparisons of experiences generated by different feedback models and rubrics generated by different policies would help clarify the improvements brought by rubric training.
- The paper emphasizes "co-evolution with policy and experience." When does this co-evolution reach a plateau, and what factors limit further improvement?
Questions for authors
- The training FLOPs x-axis in Figure 2 lacks numerical tick labels.