- Rating
- 4
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The authors use the fine-grained outcome of rubric evaluations to update the corresponding tokens for each criterion. Namely, CoCA changes advantage in GRPO loss so that at token level it is is criterion-aware. This also avoids OPSD shortcoming when imitating a self-teacher.
Critical strengths
- Very relevant and timely topic.
- I like the intuitive idea of allocating budget across tokens based on each rubric criterion and how relevant they are.
- Experimental sweep with very strong results. 4.2% improvement w.r.t. standard rubric-base RL. With higher correction and less regression.
Critical weaknesses
- None
- Minor presentation issues:
- On Equation 7, is undefined. Mention it refers to expected reward.
- Mention JSD means Jensen-Shannon divergence. After Equation 8 just write (JSD). This way its meaning is clear later on the "ablation of relevance measures" subsection.
Questions for authors
- Proposition 2 shows a possible local failure mechanism. It would be nice to further test if this is in fact the cause of the degradation for 200 training steps in Figure 1 by approximating the covariance matrices at different checkpoint to check if the covariance matrix is strictly negative.
- A similar Figure 5 or Figure 8 with OPSD assignment would be nice to have.
3 comments
Thank you for running the analysis on the covariance, it is nice to see that it agrees with your conjecture. The additional visualization will help make contribution more clear.
I maintain my rating of 4 (clear acceptance).
Thank you so much for your thoughtful follow-up and continued support! Your suggestions really helped us make this a better paper.
Thanks for your careful reading and valuable comments! We're glad that you enjoyed our work. Below are our detailed responses:
A: Thanks for pointing these out! We've clarified both points in the revised paper as you suggested.
A: Good point! We checked whether the teacher-induced matching signal tended to favor lower-reward actions during the OPSD run shown in Figure 1. At initialization and every 50 steps through step 200, we estimated the mean covariance in Proposition 2 and its 95% confidence interval.
The estimated mean covariance was negative at all five checkpoints. The 95% confidence intervals were entirely below zero at the first two checkpoints and crossed zero at the remaining three, so the evidence for reward misalignment was clearest early in training. This pattern supports the mechanism in Proposition 2 as a plausible contributor to the early reward decline in Figure 1.
A: That's a helpful suggestion! We've added this comparison in Figure 12 of the revised paper, using the same medical response and criterion as in Figure 5. Red indicates suppression and blue indicates reinforcement. The response satisfies the criterion by recommending a visit to an ophthalmologist if symptoms persist or affect vision. Yet OPSD gives negative signals to “Doctor” in the heading, “consult” in the recommendation, and “recurs” and “vision” in the conditions for seeking care.
CoCA uses prediction shifts only to identify relevant tokens; group-relative criterion outcomes determine the sign of the credit. Its relevance map in Figure 5 highlights both the advice to see a doctor and the conditions that warrant a visit. This example illustrates a limitation of OPSD and shows why credit grounding matters.
Finally, we really appreciate your kind words and support for our work. Your encouragement means a lot to us! 😊