acceptodds
Under review as a conference paper at ICLR 2027

CoCA: Criterion-Aware Credit Assignment for Rubric-Based Reinforcement Learning

16 views

Abstract

Rubric-based reinforcement learning (RL) evaluates responses against explicit criteria to construct rewards, offering a viable way to train large language models on open-ended tasks. However, standard methods compress criterion-level feedback into scalar rewards and assign uniform token advantages, leaving structured rubric supervision underutilized. Recent work seeks finer-grained guidance by treating rubrics as privileged information to derive self-distillation targets, but the resulting learning signal is not grounded in criterion outcomes and can lead to reward degradation. This motivates us to reframe the problem as criterion-aware credit assignment, where privileged rubric information serves as a relevance cue for credit allocation rather than a source of imitation targets. Accordingly, we propose **CoCA**, which derives criterion-level credits from group-relative outcomes and allocates them across tokens based on criterion-induced distribution shifts, yielding both outcome-grounded and fine-grained learning signals. Empirically, CoCA achieves an average gain of **+4.2%** over standard rubric-based RL across medicine and science benchmarks, demonstrating more effective learning on open-ended tasks.

Primary area:
foundation or frontier models, including LLMs
Keywords:
rubric-based reinforcement learning, credit assignment, large language model post-training

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.50 over 2 reviews, above 99% of the others.
Reviewer_hCcAOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The authors use the fine-grained outcome of rubric evaluations to update the corresponding tokens for each criterion. Namely, CoCA changes advantage in GRPO loss so that at token level it is is criterion-aware. This also avoids OPSD shortcoming when imitating a self-teacher.

Critical strengths
  • Very relevant and timely topic.
  • I like the intuitive idea of allocating budget across tokens based on each rubric criterion and how relevant they are.
  • Experimental sweep with very strong results. 4.2% improvement w.r.t. standard rubric-base RL. With higher correction and less regression.
Critical weaknesses
  • None
  • Minor presentation issues:
    1. On Equation 7, JJ is undefined. Mention it refers to expected reward.
    2. Mention JSD means Jensen-Shannon divergence. After Equation 8 just write (JSD). This way its meaning is clear later on the "ablation of relevance measures" subsection.
Questions for authors
  • Proposition 2 shows a possible local failure mechanism. It would be nice to further test if this is in fact the cause of the degradation for 200 training steps in Figure 1 by approximating the covariance matrices at different checkpoint to check if the covariance matrix is strictly negative.
  • A similar Figure 5 or Figure 8 with OPSD assignment would be nice to have.
3 comments

Thanks for your careful reading and valuable comments! We're glad that you enjoyed our work. Below are our detailed responses:

W1: Minor presentation issues.

A: Thanks for pointing these out! We've clarified both points in the revised paper as you suggested.

Q1: Evidence for Proposition 2's local failure mechanism.

A: Good point! We checked whether the teacher-induced matching signal tended to favor lower-reward actions during the OPSD run shown in Figure 1. At initialization and every 50 steps through step 200, we estimated the mean covariance in Proposition 2 and its 95% confidence interval.

Training stepMean covariance95% confidence interval
0-0.003666[-0.008937, -0.000471]
50-0.003742[-0.009533, -0.000199]
100-0.000576[-0.001928, +0.000754]
150-0.001976[-0.004264, +0.000402]
200-0.001158[-0.002653, +0.000012]

The estimated mean covariance was negative at all five checkpoints. The 95% confidence intervals were entirely below zero at the first two checkpoints and crossed zero at the remaining three, so the evidence for reward misalignment was clearest early in training. This pattern supports the mechanism in Proposition 2 as a plausible contributor to the early reward decline in Figure 1.

Q2: Visualization of OPSD assignment.

A: That's a helpful suggestion! We've added this comparison in Figure 12 of the revised paper, using the same medical response and criterion as in Figure 5. Red indicates suppression and blue indicates reinforcement. The response satisfies the criterion by recommending a visit to an ophthalmologist if symptoms persist or affect vision. Yet OPSD gives negative signals to “Doctor” in the heading, “consult” in the recommendation, and “recurs” and “vision” in the conditions for seeking care.

CoCA uses prediction shifts only to identify relevant tokens; group-relative criterion outcomes determine the sign of the credit. Its relevance map in Figure 5 highlights both the advice to see a doctor and the conditions that warrant a visit. This example illustrates a limitation of OPSD and shows why credit grounding matters.

Finally, we really appreciate your kind words and support for our work. Your encouragement means a lot to us! 😊

Thank you for running the analysis on the covariance, it is nice to see that it agrees with your conjecture. The additional visualization will help make contribution more clear.

I maintain my rating of 4 (clear acceptance).

Thank you so much for your thoughtful follow-up and continued support! Your suggestions really helped us make this a better paper.

Reviewer_XCBfOpenReview ↗
Rating
3
Confidence
2(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces CoCA, which separates credit anchoring from credit refinement. Credit anchoring uses observed outcomes to determine the total credit for each criterion, including its sign. Credit refinement then distributes this credit across tokens based on rubric-induced shifts in the model’s predictions. This gives CoCA fine-grained learning signals that remain tied to actual outcomes.

CoCA addresses two problems with existing methods. GRPO-style methods cannot distinguish responses that receive the same overall reward but perform differently on individual criteria. They may also reinforce tokens that violate a criterion when the overall advantage is positive. Methods such as OPSD use rubrics as privileged context to obtain token-level signals through self-distillation. However, their updates depend on differences between teacher and student predictions, not on observed criterion-level outcomes. This may hurt performance rather than improve it.

On medical and scientific benchmarks, CoCA achieves an average improvement of 4.2% over standard rubric-based reinforcement learning.

Critical strengths
  1. CoCA introduces a criterion-aware credit assignment framework for rubric-based reinforcement learning. Experiments in medical and scientific domains show that it outperforms all evaluated baselines, achieving an average improvement of 4.2% over standard rubric-based reinforcement learning.

  2. The experiments are fairly thorough and well designed. They include recent baselines, and the ablations cover the credit allocation mechanism, relevance measures, and other design choices.

Critical weaknesses
  1. β is a global hyperparameter, but the bounds in Proposition 3 differ across criteria. The authors should report how often β exceeds these thresholds in the data. Without this, it is unclear whether the guarantee holds in practice. It also seems more reasonable to adapt β for each sample rather than use one fixed value for an entire dataset.

  2. Most of the gains appear to come from non-uniform credit allocation itself, while criterion specificity contributes only approximately one-quarter of the improvement. This weakens the evidence that the criterion-aware aspect is the key driver of the method’s effectiveness.

  3. The case study should include comparisons with methods such as OPSD, rather than showing only CoCA’s behavior.

Questions for authors
  1. How are the hyperparameters K and β selected? The current approach appears heuristic and requires different settings for different datasets. Could an adaptive approach reduce the need for dataset-specific tuning?
3 comments

Thanks for your insightful comments! Our detailed responses are as follows.

W1: Empirical coverage of the β\beta bound in Proposition 3.

A: We computed the bound Bij=8Δij/bij2B_{ij}=8\Delta_{ij}/b_{ij}^{2} for each response–criterion pair in our analysis and compared it with the default β\beta for that domain. The default β\beta falls below the bound for 95.7% of the pairs in Medicine and 100% in Science. The table below summarizes the results.

Domainβ\betaMean BijB_{ij}Median BijB_{ij}Min. BijB_{ij}Pairs with β<Bij\beta < B_{ij}
Medicine572.4516.804.6195.7%
Science3684.92106.596.82100.0%

Both the mean and median bounds are well above the default β\beta in each domain. Even the minimum bound exceeds β\beta in Science and falls only slightly below it in Medicine. These results show that the guarantee typically holds in practice. We also report results for adaptive β\beta in Q1 below.

W2: Contribution of criterion-specific allocation.

A: Actually, criterion specificity accounts for most of CoCA's improvement. The ablation results in Table 2 show this directly. Shared distributes the response-level advantage non-uniformly across tokens without allocating credit separately for each criterion. This yields only a 1.4-point gain over Uniform (60.3 to 61.7).

By allocating credit separately for each criterion, CoCA reaches an average score of 64.5, 4.2 points above Uniform and 2.8 points above Shared. This 2.8-point gain over Shared represents two-thirds of the total improvement over Uniform. The comparison highlights the value of criterion-specific credit allocation.

W3: Comparison with OPSD in the case study.

A: That's a helpful suggestion! Figure 12 of the revised paper shows OPSD's token-level signal for the same medical response and criterion as Figure 5. Red indicates suppression and blue indicates reinforcement. The response satisfies the criterion by advising the patient to consult an ophthalmologist if symptoms persist or vision changes. Yet OPSD gives negative signals to “Doctor” in the heading, “consult” in the recommendation, and “recurs” and “vision” in the conditions for seeking care.

In contrast, CoCA uses criterion-induced prediction shifts only to assess token relevance; group-relative criterion outcomes determine the sign of the credit. In Figure 5, this relevance signal correctly highlights both the advice to see a doctor and the conditions that warrant a visit. This example illustrates a limitation of OPSD and shows why credit grounding matters.

Q1: Could kk and β\beta be chosen adaptively to reduce dataset-specific tuning?

A: That's an interesting idea to explore! In the paper, we chose kk and β\beta empirically based on the results in Figure 4(b). We see the two choices a little differently: kk lets users decide how much computation to spend, while β\beta controls how concentrated the token weights are.

(1) For kk, a larger value lets CoCA allocate credit separately for more criteria, but each additional criterion requires another forward pass. We use k=2k=2 in both domains because larger values yield only marginal gains. We'd therefore leave kk as a hyperparameter that users set before training to balance cost and performance.

(2) For β\beta, it makes sense to adapt the strength of credit allocation to each response. As a first step, inspired by Sun et al. [1], we set a response-specific βi\beta_i inversely to the spread of JSD scores: βi=(1∣Si∣∑j∈SiVar⁡t(dij,t))−1/2\beta_i=\left(\frac{1}{|S_i|}\sum_{j\in S_i}\operatorname{Var}_t(d_{ij,t})\right)^{-1/2}. When the scores vary widely across tokens, a smaller βi\beta_i tempers the differences in their weights; when the scores are close, a larger βi\beta_i makes those differences more pronounced. This lets CoCA allocate credit based on how much each token stands out within the response, without tuning β\beta separately for each domain. We compare adaptive and fixed β\beta below.

SettingMedicine avg.Science avg.
Fixed β=3\beta=364.362.7†
Fixed β=5\beta=565.9†61.9
Adaptive βi\beta_i64.562.9

† denotes the original setting for that domain. Table 10 of the revised paper gives the full results.

Adaptive βi\beta_i achieves the highest average score in Science among the settings tested. In Medicine, it improves on fixed β=3\beta=3 but falls short of the original β=5\beta=5 setting. We find this encouraging: using the same adaptive rule across domains could reduce the need for dataset-specific tuning and, in some cases, even outperform the best fixed setting.

[1] Logit Standardization in Knowledge Distillation. CVPR'2024.

Thanks again for your thoughtful comments and suggestions! They helped us improve the paper and gave us new insights. 😊

Thank you to the authors for their rebuttal. I have no further questions.

Thank you for taking the time to review our paper and for getting back to us. We’re very glad our response addressed your concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.