acceptodds
Under review as a conference paper at ICLR 2027

Rubric-to-Segment: Criterion-grounded Semantic Credit Assignment for Language Model Alignment

Abstract

Rubric-based reinforcement learning translates open-ended alignment objectives into structured, interpretable supervision through explicit evaluation criteria. However, standard rubric-based group relative policy optimization (GRPO) aggregates criterion scores into a single reward and broadcasts the resulting advantage across the response, obscuring criterion-specific performance and its correspondence to response content. Our analysis of safety and medical rubrics and model responses finds that most criteria concern specific semantic content, with evidence often localized to different parts of a response. Building on this correspondence, we introduce Rubric-to-Segment, which turns structured rubric judgments into contentspecific training feedback by linking criterion-wise comparison across responses with evidence-based allocation within each response. Using scores and evidence from a single judge assessment, it computes group-relative advantages separately for each criterion and allocates them to semantic segments as local corrections to the response-level advantage. Across two backbones in safety and medical alignment, Rubric-to-Segment improves task performance over response-level rubric GRPO-based and reinforcement learning–distillation baselines while maintaining general capabilities.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM Alignment, Safety

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_wvvdOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

Rubric-to-Segment (RtS) addresses coarse credit assignment in rubric-based GRPO. It compares sampled answers separately on each criterion and uses judge-cited evidence to add segment-level corrections to the global advantage. On safety and medical tasks with two Qwen models, the paper reports gains over rubric GRPO and RLSD, plus ablations and a Dr. GRPO extension.

Critical strengths
  1. RtS turns criterion scores and evidence citations from one judge assessment into a simple local update. The paper also shows that equal total rewards can hide different criterion scores and that criteria often point to different parts of an answer (Figure 2; Appendices C–D).

  2. RtS improves the reported task score over GRPO and RLSD for both models in both domains (Table 1). The ablations and Dr. GRPO results add useful support, while general-capability scores show no broad decline (Tables 2–3).

Critical weaknesses
  1. The shuffled-evidence control does not isolate whether feedback reaches the right content. Equation 5 centers corrections across segments, but the loss applies them to tokens; shuffling evidence across unequal-length segments can therefore change the answer's mean token advantage as well as its location. If the Table 2 gap persists after matching that mean, the semantic-localization claim would be more convincing.

  2. The comparison is broad, covering GRPO, RLSD, Dr. GRPO, and even OPD, but it omits a matched multi-reward, response-level method such as GDPO. The Table 2 ablations retain local allocation or shuffle its evidence, so they do not show how much of the gain comes from criterion-wise normalization alone. If RtS remains stronger than a response-level combination of the same criterion advantages under the same training budget, this concern would be resolved.

Questions for authors
  1. The following question is related to my concern in Weakness 1. How large is the token-mean shift caused by shuffling? Can a more controlled ablation separate this length effect from semantic correspondence, and does RtS still win?

  2. As a complementary test to shuffling, does RtS outperform a response-level combination of the same criterion-wise advantages with matched rubric scores and training budget?

Reviewer_UBVCOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to use rubric feedback more effectively in LLM alignment task. Standard rubric based GRPO combine cirterion scores into one reward scalar and assigns the same advantage to every token in one response. In this paper, rubric-to-segment instead seperately normalizes each criterion across the group of responses, then identifies the relevant segments by using the judge-cited evidence, and finally adds the segment level corrections to the original response advantage. This paper conducts experiments on safety and medical alignment tasks with Qwen2.5-7B and Qwen3-8B and shows improvements over GRPO and RLSD.

Critical strengths

I find the motivation of this paper convincing. Two responses could receive the same overall reward scalar while doing well on different requirement, and different segments in a response deserve different feedbacks. Therefore, using the criterion to comparing different responses makes sense to me and designing the segment level correction added to the response level advantage is also reasonable. I also like that the method is relatively simple to implement: it gets the criterion scores and evidence from a singe judge assessment without additionally training a attribution model and the designed advantage can be directly used in the standarad GRPO objective function.

Critical weaknesses

In 'Strengths' part, I mentioned both two parts are intuitively reasonable. However, I am not fully convinced by the current experiments for the reasons as follows.

First, this paper lacks a direct comparison with GDPO [1], which is also listed by the authors in related work. GDPO already performs criterion-wise normalization and using the resulting advantages in the response level. This paper provides aggregate-advantage and shuffled-evidence ablations, but it lacks the direct comparison with this simpler alternative and still remain unclear how much additional benefit comes from segment level credit assignment. I think this additional comparison is necessary and would like to see the results.

Second, the reliability of evidence localization is not directly evaluated. table 5 only shows that ,with evidence attribution added, the aggregate scores remain consistent, but this does not indicate whether the cited segments are correct. Although the shuffled-evidence ablation provides some support, it does not measure localization accuracy. Because the method relies on assigning criterion-wise feedback to corresponding segments, I think a direct evaluation of evidence localization would strengthen the paper.

[1] Liu, Shih-Yang, et al. Gdpo: Group reward-decoupled normalization policy optimization for multi-reward rl optimization.

Questions for authors

See 'Weaknesses' part.

  1. Could you add a GDPO response-level baseline using the same criterion scores and a comparable advantage scale?

  2. How do the authors evaluate the correctness of evidence localization and how sensitive is RtS to errors in evidence localization?

Reviewer_g5SrOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Rubric-based GRPO sums criterion scores into one reward and broadcasts the resulting advantage over the whole response. RtS changes two things. Each criterion is normalized separately across the rollout group, and the judge (gpt-oss-120b) returns, in the same call, the sentence or list-item segments supporting each score. Each segment's advantage is the response-level advantage plus a centered, rubric-weighted sum of the criterion advantages routed to it (Eq. 5). The method is trained on Qwen2.5-7B-Instruct and Qwen3-8B in a safety domain and on RaR-Medicine, compared with rubric GRPO, RLSD and Dr. GRPO, and ablated by replacing criterion advantages with the aggregate one or by shuffling evidence across segments. RtS has the best Safety composite and HealthBench overall score in every setting, while the four-benchmark general-capability average stays roughly where it was.

Critical strengths
  1. The gains are consistent under matched training budgets. RtS shares queries, judge, rollouts and optimizer settings with GRPO, yet improves XSTest by 4.8 points on both backbones while HarmBench and WildJailbreak stay above 95, and moves Qwen2.5 HealthBench from 23.31 to 27.18. Added to Dr. GRPO, it raises the Safety composite from 91.4 to 95.1 and from 90.3 to 93.8, and HealthBench from 24.75 to 27.55 and from 35.29 to 35.97. No single margin is decisive, but eight comparisons in the same direction are hard to dismiss.

  2. The method costs almost nothing extra. Scores and evidence come from one judge call, no attribution model or teacher is trained, and the change is confined to the advantage computation. It can be tried on top of any rubric-reward pipeline, which matters given how widely rubric rewards are now used.

Critical weaknesses
  1. The source of the gain is not pinned down. There are two factors (aggregate vs. criterion-wise comparison, broadcast vs. localized allocation), and only three of the four cells are run. The missing one, criterion-normalized advantages applied at the response level as in GDPO, is the closest competitor. RtS has no such component: with uniform evidence weights Eq. 5 collapses to bib_i, so criterion-wise comparison and localization cannot be separated. The centering also behaves oddly. Segments a criterion does not cite receive a correction of the opposite sign; in Figure 3, the poor score on C4 (a=−1.00a=-1.00, evidence S4–S5) adds positive credit to S1–S3. This conflicts with the stated motivation that content unrelated to a judgment should not receive its feedback, and the shuffled control keeps the same centering, so it cannot test this.

  2. The evidence is thinner than the tables suggest. No run-to-run variation is reported, and each benchmark is evaluated with two samples per item. Several margins are small: on Qwen3 medicine RtS beats RLSD by 0.42 and GRPO by 0.69, and both ablations fall below plain GRPO there (34.49 and 33.56 vs. 34.80); on Qwen2.5 safety the aggregate-advantage variant ties RtS at 94.0. The general-capability numbers point to a noise floor of several points. Safety RL moves Qwen3 GPQA from 43.7 to 51.5 and separates GRPO from RtS on Qwen2.5 GPQA by 5.3 points, which is hard to credit to the training signal. The averages also hide IFEval drops after safety training (71.6 to 63.0 and 82.8 to 74.0). Localization accuracy is never measured: Appendix A.3 compares scores with and without evidence, and does so with Qwen3.5-397B rather than the gpt-oss-120b judge that produced the training evidence.

Questions for authors
  1. Can you add the response-level criterion-normalized baseline (bi+λ∑dαdaidb_i + \lambda\sum_d \alpha_d a_i^d applied to all tokens) and a variant in which uncited segments keep bib_i without centering? These two runs would show whether localization itself is doing the work.

  2. Can you report at least three seeds for GRPO, RtS and the two ablations in one setting, preferably Qwen3 on RaR-Medicine where the margins are smallest?

  3. How often does a criterion fall back to uniform weights (no evidence, unpositioned omission, or more than 60% coverage), and how accurate are the gpt-oss-120b citations under a human audit, reported separately for expressed content and positioned omissions?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.