acceptodds
Under review as a conference paper at ICLR 2027

Alternating Reinforcement Learning with Contextual Rubric Rewards: Beyond the Scalarization Strategy

Abstract

Reinforcement Learning with Rubric Rewards (RLRR) is a framework that extends conventional reinforcement learning from human feedback (RLHF) and verifiable rewards (RLVR) by replacing scalar preference signals with structured, multi-dimensional contextual rubric-based evaluations. However, existing approaches in RLRR are limited to linearly compress vector rewards into a scalar reward with fixed weightings, which is sensitive to artificial score design and fails to capture correlations among reward dimensions. To overcome the limitations of reward aggregation, this work proposes Alternating Reinforcement Learning with Rubric Rewards (ARL-RR), a framework that removes the need for a fixed scalarization by optimizing one semantic rubric meta-class at a time. Theoretically, we show a variance contraction effect of reward aggregation that explains the performance improvement. We further introduce a lightweight search-based adaptation procedure that selects the next meta-class dynamically based on task performance, enabling the policy to emphasize critical objectives and thus improve the model performance. Empirically, our experiments on the HealthBench dataset with experts' annotation demonstrate that ARL-RR uniformly outperforms scalarized methods in both model performance and training efficiency across different model scales (1.7B, 3B, 8B, and 14B).

Primary area:
foundation or frontier models, including LLMs
Keywords:
Reinforcement Learning, Post-Training, Large Language Model

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_k7nuOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Alternating Reinforcement Learning with Rubric Rewards (ARL-RR), a method for optimizing large language models using contextual, multi-dimensional rubric feedback without collapsing all criteria into a single scalar reward. The method maps task-specific rubric criteria into five semantic meta-classes—accuracy, completeness, instruction following, context awareness, and communication quality—and alternates policy optimization across these classes. It further introduces a search-based procedure that selects the next meta-class according to intermediate task performance. The paper also provides a theoretical analysis of variance contraction under scalarized rewards and evaluates the proposed approach on HealthBench using several actor model sizes and reinforcement learning algorithms.

Critical strengths
  1. Relevant problem formulation. The paper addresses a meaningful limitation of scalarized reward optimization. A weighted average can allow strong performance on some rubric dimensions to compensate for poor performance on others, even when individual criteria are important in their own right. Optimizing rubric meta-classes separately is a straightforward and potentially useful alternative that preserves criterion-level distinctions during policy updates.
  2. Simple and practical optimization framework. ARL-RR builds on an existing GRPO-style training framework and changes the reward used at each training stage rather than requiring a fundamentally different policy architecture. The semantic meta-class representation also provides an interpretable interface between contextual rubrics and the optimization process. The supplementary experiments on synthetic meta-classification suggest that the approach may remain useful when expert-provided meta-class labels are unavailable, although some performance is lost relative to expert labels.
Critical weaknesses
  1. The theoretical analysis does not fully justify the claimed optimization advantage. The variance-contraction theorem in Section 3.2 and Appendix B assumes that all meta-class rewards have the same variance and that their pairwise covariances are bounded by a common correlation parameter. Under these assumptions, the variance of their weighted average is smaller than the variance of an individual meta-class reward. The mathematical argument is reasonable under the stated assumptions, but the assumptions are restrictive: different rubric dimensions can have substantially different reward distributions, and their correlations need not follow the assumed structure. More importantly, showing that scalarization reduces reward variance does not, by itself, establish that scalarized RL has a weaker policy-learning signal or that alternating optimization should perform better. The paper uses GRPO, where advantages are normalized within a rollout group. A uniform reduction in reward scale can be largely canceled by this normalization. The more relevant question is whether scalarization changes the relative ordering of responses, the resulting normalized advantages, or the alignment between policy gradients and the desired objectives.
  2. The novelty and positioning relative to existing multi-objective RL methods need stronger justification. The central algorithmic idea is to decompose contextual rubric criteria into semantic groups and alternate policy optimization across those groups. This is intuitive, but the paper does not sufficiently establish which benefits specifically arise from the proposed alternating strategy rather than from optimizing one reward component at a time more generally. The paper should clarify what is fundamentally new about applying alternating optimization to contextual rubrics, beyond the particular choice of five semantic meta-classes and the proposed schedule search.
Questions for authors

Please see the weaknesses.

Reviewer_u6SLOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ARL-RR groups contextual rubric criteria into five semantic categories and trains by alternating across category-specific rewards rather than collapsing them into a single scalar reward. The paper also considers a search variant that chooses the next category through short candidate training runs. A variance-based argument is used to motivate avoiding fixed reward aggregation. On HealthBench, the fixed-order ARL variant outperforms scalarized RL across four Qwen3 model sizes while requiring less time per optimization step. The search-based version is evaluated separately with Qwen3-4B.

Critical strengths

1.The method is simple and produces consistent gains at lower training cost. Across all four Qwen3 scales in Table 1, ARL improves over SRL while using the same number of prompts and optimization steps. The reported results are averaged over five seeds, and the per-step runtime is substantially lower. Since ARL mainly changes which rubric criteria contribute to the reward at each stage, the method could be easy to integrate into existing rubric-based RL pipelines.

2.The empirical trend extends beyond the main setting. The gains remain visible with Llama and Mistral actors and under GSPO and DAPO. Table 4 also shows that, when evaluated by the common 32B judge, ARL remains better than SRL after training with smaller 4B, 8B, and 14B reward models. This gives some evidence that the effect is not limited to a single actor or optimization algorithm.

Critical weaknesses

1.Can the authors define the exact reward used by SRL and ARL more precisely? The current theory is based on contraction of raw reward variance under assumptions such as positive criterion weights and comparable class variances. However, GRPO normalizes rewards within each group, so larger raw variance does not directly imply a stronger learning signal after normalization. In addition, HealthBench contains negative rubric points, while the theoretical presentation appears to assume positive weights. Table 7 also suggests that many prompts do not contain every rubric class, but it is unclear how missing classes are handled. Please clarify the signed reward definition, any transformation or clipping, the treatment of absent classes, and the effective number of prompts and rollouts used for each class. This is mainly a specification and interpretation issue rather than evidence that the implementation is incorrect.

2.How robust are the reported gains to evaluation choices and uncertainty? The main experiments use Qwen3-32B both as the training judge and as the primary evaluator. This makes the SRL-versus-ARL comparison internally consistent, but it leaves open whether some gains are judge-specific. The reported standard errors also appear difficult to interpret, especially the difference between the 0.08–0.18 values and the 1.67 value for the 14B model. For smaller gains such as the 8B result, paired seed-level differences and confidence intervals would make the evidence much easier to assess. An independent judge or a small blinded human re-evaluation of existing checkpoints would also strengthen the result without requiring retraining.

Questions for authors

See weakness.

Reviewer_HuxAOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses a genuine limitation of Reinforcement Learning with Rubric Rewards (RLRR): the standard practice of linearly compressing a vector of rubric rewards into a scalar via fixed weights. The ARL-RR (Alternating Reinforcement Learning with Rubric Rewards) is proposed.

Critical strengths

Theorem 3.1 is a variance-contraction result with a straightforward proof. Section 4.3 (Table 2) provides an empirical check: scalarized reward variance 0.10 vs meta-class variances 0.16–0.20, computed on a frozen model with matched rollout settings.

Critical weaknesses

The results are on HealthBench. Evaluation on an open-domain instruction-following or chat benchmark is needed before generalization claims are credible. The authors have acknowldeged the dataset narrowness, synthetic-label cost, and dependence on reward-model fidelity.

Questions for authors

In the meta-class searching procedure, what data is used to evaluate checkpoints and select the winning meta-class? Is the 1K evaluation set involved at any point in training decisions?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.