- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Alternating Reinforcement Learning with Rubric Rewards (ARL-RR), a method for optimizing large language models using contextual, multi-dimensional rubric feedback without collapsing all criteria into a single scalar reward. The method maps task-specific rubric criteria into five semantic meta-classes—accuracy, completeness, instruction following, context awareness, and communication quality—and alternates policy optimization across these classes. It further introduces a search-based procedure that selects the next meta-class according to intermediate task performance. The paper also provides a theoretical analysis of variance contraction under scalarized rewards and evaluates the proposed approach on HealthBench using several actor model sizes and reinforcement learning algorithms.
Critical strengths
- Relevant problem formulation. The paper addresses a meaningful limitation of scalarized reward optimization. A weighted average can allow strong performance on some rubric dimensions to compensate for poor performance on others, even when individual criteria are important in their own right. Optimizing rubric meta-classes separately is a straightforward and potentially useful alternative that preserves criterion-level distinctions during policy updates.
- Simple and practical optimization framework. ARL-RR builds on an existing GRPO-style training framework and changes the reward used at each training stage rather than requiring a fundamentally different policy architecture. The semantic meta-class representation also provides an interpretable interface between contextual rubrics and the optimization process. The supplementary experiments on synthetic meta-classification suggest that the approach may remain useful when expert-provided meta-class labels are unavailable, although some performance is lost relative to expert labels.
Critical weaknesses
- The theoretical analysis does not fully justify the claimed optimization advantage. The variance-contraction theorem in Section 3.2 and Appendix B assumes that all meta-class rewards have the same variance and that their pairwise covariances are bounded by a common correlation parameter. Under these assumptions, the variance of their weighted average is smaller than the variance of an individual meta-class reward. The mathematical argument is reasonable under the stated assumptions, but the assumptions are restrictive: different rubric dimensions can have substantially different reward distributions, and their correlations need not follow the assumed structure. More importantly, showing that scalarization reduces reward variance does not, by itself, establish that scalarized RL has a weaker policy-learning signal or that alternating optimization should perform better. The paper uses GRPO, where advantages are normalized within a rollout group. A uniform reduction in reward scale can be largely canceled by this normalization. The more relevant question is whether scalarization changes the relative ordering of responses, the resulting normalized advantages, or the alignment between policy gradients and the desired objectives.
- The novelty and positioning relative to existing multi-objective RL methods need stronger justification. The central algorithmic idea is to decompose contextual rubric criteria into semantic groups and alternate policy optimization across those groups. This is intuitive, but the paper does not sufficiently establish which benefits specifically arise from the proposed alternating strategy rather than from optimizing one reward component at a time more generally. The paper should clarify what is fundamentally new about applying alternating optimization to contextual rubrics, beyond the particular choice of five semantic meta-classes and the proposed schedule search.
Questions for authors
Please see the weaknesses.