acceptodds
Under review as a conference paper at ICLR 2027

RAY-GRPO: MULTI-OBJECTIVE ALIGNMENT THROUGH RAY-GUIDED REWARD WEIGHTING

Abstract

Group Relative Policy Optimization (GRPO) learns from within-group reward comparisons. Multiple objectives require reward credit that adapts to the policy’s changing bottlenecks. We introduce RAY-GRPO, with two components: raybased preference smoothing and policy-dependent reward reweighting. The first balances reference directions across controls and averages exact radial utilities, assessing several reward proportions at the same attained rewards. The second differentiates this average, giving each objective credit through the directions it currently limits. Auxiliary rays reuse observed rewards. Cross-fitted coefficients and alternating updates to endpoint low-rank adapters and a gated residual train one conditional policy while preserving endpoint behavior at vertices. The associated surrogate admits a finite-ray hypervolume lower bound and nonzero direct credit for strictly positive control means. Experiments with two, three, and five objectives demonstrate gains in overall reward and empirical Pareto-front quality against the evaluated baselines.

Primary area:
reinforcement learning
Keywords:
GRPO, Multiobjective alignment

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_a3LtOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

RAY-GRPO adapts the reward coefficients used by GRPO according to both the requested preference and the policy's current reward attainment. For each nonvertex control, balanced transport assigns a distribution of reference rays in calibrated reward space. The method measures attainment along each ray by its limiting reward-to-proportion ratio and differentiates the resulting utility to give more credit to objectives that currently limit the desired balance. It also alternates updates to vertex-specific LoRA blocks with updates to a control-conditioned residual for nonvertex trade-offs. The experiments cover multi-objective alignment; the clearest gains are on the seven two-objective tasks, while frontier-coverage results are more mixed in the other settings.

Critical strengths
  • The reward-allocation rule has a clear interpretation: a ray is limited by the objective with the smallest attained-reward-to-required-proportion ratio. The derived coefficients reuse scores from existing rollouts, so the method does not require extra generations or reward-model evaluations.
  • The boundary/interior parameterization is carefully designed. During nonvertex refinement, the endpoint blocks are frozen and the residual vanishes at simplex vertices, so those updates leave the current endpoint policies unchanged.
  • The empirical study is broad. It includes thirteen methods on seven two-objective tasks, transfer evaluation with three objectives, a five-objective setting, ablations, and HH2 results on two additional backbones. The appendix is unusually explicit about configurations and comparison limitations.
Critical weaknesses
  • Training and evaluation rely on the same frozen reward models. The paper itself appropriately describes the results as controlled reward-model measurements, but held-out evaluators or human preferences are needed before interpreting them as broader alignment improvements.
  • The three-objective comparison is exploratory because the reported evaluations were also used to select the ray count. The five-objective experiment covers only three controls on a restricted simplex path, so it cannot establish dense preference coverage or responsiveness over the full simplex.
  • Baseline parity is incomplete. Appendix C notes differences or missing evidence concerning truncation, prompt grouping, LoRA capacity, update and response budgets, GPU time, and historical inputs. These caveats make some of the cross-method gains difficult to attribute solely to RAY-GRPO.
  • No variance across independent training runs is reported. In addition, the joint mechanism-removal ablation changes the adapter structure, endpoint credit, update schedule, and KL reference together, so it does not isolate which part of the boundary/interior design matters most.
Questions for authors

Please refer to the Weaknesses section.

4 comments
AuthorsOpenReview ↗

Reply to a3Lt (1/3)

We organize the response around the four concerns in your Weaknesses section.

Concern 1. Given the same frozen reward models for training and evaluation, do held-out evaluators or human preferences support broader alignment claims?

Yes. Two raters compared final RayGRPO and GDPO policies across three controls with model identities masked, covering 135 distinct HH2 prompts and 270 main judgments. All main judgments are retained. Wins/ties/losses score 1/0.5/0. Evaluation CIs use 10,000 resamples of retained prompts with replacement, keeping policies, answers and judgments fixed. Combined results are:

Human evaluationCombined result
Overall RayGRPO preference score [95% CI]60.2% [53.3,66.9]
Helpfulness59.6% [52.8,66.1]
Harmlessness59.8% [54.1,65.4]

The combined blinded ratings favor RayGRPO over GDPO in Overall preference, with Helpfulness and Harmlessness judgments pointing in the same direction. This provides human evidence that RayGRPO's advantage extends to perceived response quality in the evaluated HH2 sample.

GPT-5.4 (gpt-5.4-2026-03-05) also assessed model-masked answers in both orders for 1,000 RayGRPO response samples spanning 428 distinct prompts and 11 controls. The evaluation retains the earlier 90 samples and judgments; 15 historical pairs were excluded for uncertain input matching. Earlier judgments preceded expansion and retain a separate analysis target. The same answer-completeness rule is applied to every method.

The comparison includes 1,000 GDPO pairs and 995 pairs for each other baseline. HoE and Soups each have 994 complete pairs for Overall and Helpfulness, and 995 for Harmlessness. Scores use the same win/tie/loss rule, average both answer orders, and weight evaluation groups equally. Evaluation CIs use 10,000 resamples of retained prompts with replacement, keeping each prompt's comparisons together and policies, answers and judgments fixed.

RayGRPO preference scores are reported with 95% pointwise confidence intervals.

ComparatorOverallHelpfulnessHarmlessness
GDPO58.1% [55.2,61.0]58.0% [55.1,60.9]53.5% [52.3,54.7]
Linear-GRPO63.0% [60.4,65.6]63.0% [60.4,65.6]53.0% [51.8,54.2]
HoE60.7% [57.9,63.7]60.7% [57.9,63.6]53.3% [52.1,54.6]
Rewarded Soups61.6% [58.6,64.7]61.6% [58.6,64.6]54.0% [52.6,55.4]

All lower bounds remain above indifference when missing orientations count as losses.

Across the evaluated responses, the judge favors RayGRPO over every baseline in Overall, Helpfulness and Harmlessness, with larger preference margins for Helpfulness and all pointwise intervals above indifference. Together with the blinded human preference for RayGRPO over GDPO, this supports a response-quality advantage beyond the training reward models.

Concern 2. Does selecting K3 ray count on the reported evaluations bias the results, and can K5’s three-control restricted path establish full-simplex coverage or responsiveness?

K3: transfer and ray-count selection. RayGRPO transfers useful reward trade-offs to Personalized-Soups (PSoups) and HelpSteer2 without training on either target domain. At the reported M=128 setting, it improves both Helpfulness and Humor over GDPO on both datasets, so the gain is not solely a consequence of Humor's scale. The full retained ray-count comparison is:

DatasetSettingHelpfulness / Harmlessness / HumorOverallnHV
PSoupsRayGRPO, M=128-0.9038 / 1.1621 / 3.07771.11200.9022
PSoupsRayGRPO, M=512-0.8369 / 1.0590 / 2.71790.98000.9376
PSoupsRayGRPO, M=2048-0.8571 / 0.9729 / 2.85110.98900.9282
PSoupsGDPO-1.2687 / 1.2844 / 2.73590.91720.9170
HelpSteer2RayGRPO, M=128-0.0894 / 0.1493 / 2.20750.75580.8380
HelpSteer2RayGRPO, M=5120.1126 / 0.1417 / 1.60780.62070.8395
HelpSteer2RayGRPO, M=20480.0003 / -0.0358 / 1.85680.60710.8333
HelpSteer2GDPO-0.4088 / 0.2749 / 1.50380.45660.8413

All retained settings exceed GDPO in Overall on both domains, including the former default M=512, which retains its earlier calibration. The advantage therefore persists beyond the selected setting. These transfer evaluations informed the choice of M; they do not constitute independent selection and testing. Overall averages raw rewards and remains scale-dependent, whereas nHV measures frontier volume: GDPO's higher nHV at the reported setting is compatible with RayGRPO's better Helpfulness and Humor.

K5: control beyond the reported path. We analyzed an existing expanded evaluation of the submitted RayGRPO K5 checkpoint on 43 controls × 503 prompts per control. It covers simplex vertices, equal-weight faces and asymmetric interior weights, extending beyond the manuscript's three path controls.

AuthorsOpenReview ↗

Reply to a3Lt (2/3)

Concern 2 (continued).

Steerability is the correlation between each objective's weight and its mean raw reward; the mean and worst values summarize the five objectives. Nondominated counts exclude control means dominated by another tested control; the single-box share measures how much total HV one control alone covers.

Expanded K5 diagnosticResult
Mean / worst steerability0.2867 / 0.2354
Nondominated control means4
HV accounted for by largest single control box99.9%

The positive correlations show a modest control response beyond the original path, while the nondominated-count and volume-concentration diagnostics show that the frontier remains concentrated. We interpret K5 as evidence of achievable operating points, with preference responsiveness assessed separately from frontier volume. The expanded evaluation uses its own protocol and has no baseline comparison on this grid.

Concern 3. Are baselines comparable in input truncation, prompt grouping, LoRA capacity, generated-response/update budgets, GPU time and historical effective inputs?

On-policy HH2 comparisons. The historical GDPO and Linear-GRPO runs already matched RayGRPO's sampling and update budgets. Each original prompt produces two responses at each of four controls; eight prompt draws therefore yield 64 responses per optimizer update. With one PPO pass per batch, each method uses 40,016 prompt draws, 320,128 generated responses and 5,002 updates. All three have 46.01M adapter parameters. GDPO's normalization reuses the same responses; different rank counts and microbatch sizes preserve these totals. Thus, the original comparison matches prompt and generated-response exposure as well as the update-to-sample ratio.

The HH2 MO-GRPO baseline also matches these prompt, response and update totals, but samples eight responses at one control per prompt. For K3 and K5, MO-GRPO matches RayGRPO's generated-response/update budgets: respectively 96 responses per update for 5,002 updates and 160 for 1,181 updates. Its grouping gives it more prompt draws, and its adapter capacity is larger. These budgets preserve generated-response exposure and update frequency across different parameterizations.

Expert-based and additional baselines. Rewarded Soups trains two objective-specific experts, each with 10,005 updates and 16 responses per update, totaling 320,160 responses across both experts. This closely matches HH2's generated-response budget, but its optimizer-update count differs. HoE reuses those experts and adds router training and selection; these are additional costs. Other method-specific comparisons retain their documented training recipes: for example, the additional K3 MORLHF/PPO baseline uses its step-1,200 checkpoint with 9,600 responses. We do not describe these different training procedures as sharing one update-to-sample ratio.

Direct RayGRPO–GDPO comparison. We retained the historical sampling and update budgets in the additional three-seed HH2 study while standardizing the boundary/interior schedule, rank count and microbatching. The final evaluation also uses identical effective input tokens, controls and greedy decoding. Equal accelerator-time limits would not ensure equal training exposure across execution configurations, so we report measured training and evaluation costs separately. The following costs average the three seeds; performance and attribution results follow in Concern 4.

Cost metricRayGRPOGDPO
Mean training accelerator-hours208.72210.57
Mean evaluation accelerator-hours7.067.02

These costs cover retained training and evaluation. The standardized HH2 study establishes input parity for this comparison; historical K3/K5 effective-input parity is not inferred from matching prompt lists alone.

Concern 4. Can independent training runs quantify variance, and which parts of the boundary/interior design matter when the ablation changes adapters, endpoint credit, update schedule and KL reference together?

Three training runs per method. We trained RayGRPO and our main comparator, GDPO, separately with three different random seeds each. The study retains the budgets in Concern 3 and standardizes architecture, boundary/interior schedule, rank count and microbatching. All trained policies were evaluated on the same 512 prompts × 11 controls with greedy decoding. RayGRPO exceeds GDPO in nHV and Overall for each training seed.

Overall averages raw rewards across objectives and controls; steerability is the objective-wise Spearman correlation between preference weight and raw reward. Overall and nHV report mean ± SD across training runs; the remaining columns average the training runs, with worst steerability taking the lower objective correlation within each seed before averaging.

AuthorsOpenReview ↗

Reply to a3Lt (3/3)

Concern 4 (continued).

MethodMean Helpfulness / HarmlessnessOverallnHVMean / worst steerability
RayGRPO1.0221 / 0.81520.9187 ± 0.03951.0558 ± 0.01930.9939 / 0.9879
GDPO1.3391 / 0.22220.7807 ± 0.04200.7901 ± 0.01660.9652 / 0.9485

Isolating interior credit. We also ran a two-by-two comparison of credit rule and training design, with three separate training runs in every cell. The split design above uses separate endpoint adapters and an interior residual, alternating boundary/interior updates, and an endpoint-based interior KL reference. The joint design uses one preference-conditioned adapter, joint updates, and a fixed base-model KL reference. Both designs retain pure-objective endpoint credit.

Within each design, Fixed credit GRPO replaces only the interior ray-derived coefficients with w/2, where w is the requested preference vector. Architecture, schedule, KL-reference rule, leave-one-out reward baseline, root-mean-square normalization and generated-response/update budgets are held fixed. The table uses the same summary definitions as above:

Design / creditMean Helpfulness / HarmlessnessOverallnHVMean / worst steerability
Split / RayGRPO1.0221 / 0.81520.9187 ± 0.03951.0558 ± 0.01930.9939 / 0.9879
Split / Fixed credit GRPO0.3383 / 1.45940.8988 ± 0.47871.0811 ± 0.08630.7500 / 0.6788
Joint / RayGRPO1.2140 / 0.03770.6259 ± 0.00580.6583 ± 0.00090.7470 / 0.5879
Joint / Fixed credit GRPO1.1806 / 0.04910.6149 ± 0.00280.6586 ± 0.00380.7985 / 0.6818

To make conditioning reliability visible, the next table reports each split-design seed separately. Each entry gives mean / worst objective-wise steerability:

Training seedRayGRPOFixed credit GRPO
00.9955 / 0.99090.9818 / 0.9727
10.9955 / 0.99090.3591 / 0.2182
20.9909 / 0.98180.9091 / 0.8455

Fixed credit GRPO is more vulnerable to partial preference-conditioning collapse in the observed split-design runs, while RayGRPO retains near-perfect ordering in every seed. Fixed credit GRPO's higher mean split nHV shows why frontier volume and reliable preference control should be assessed separately. The joint credit contrast has nearly equal nHV. Thus, the evidence supports ray credit's role in reliable control within the proposed design; within-design contrasts isolate the coefficient rule, while comparisons across designs assess the coordinated architecture, schedule and KL choices.

Dear Reviewer a3Lt,

Thank you again for your careful review. We have provided additional experiments and clarifications in response to your comments, and would appreciate your thoughts when you have a chance to revisit them. Please let us know if any concerns remain or if further explanation would be helpful. We would be glad to continue the discussion.

Thank you for your time and consideration.

Reviewer_gLNHOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes RAY-GRPO for multi-objective GRPO. It trains one model that takes a preference vector as input. Instead of fixed reward weights, it sets the weights from the current rewards, so the reward that is falling behind gets more weight. The weights come from the ray form of hypervolume. RAY-GRPO has the highest nHV on all 7 two-objective tasks.

Critical strengths
  1. The weights are adaptive to the current rewards. The reward that falls behind gets more weight, and this weighting comes from hypervolume. It reuses scored rollouts, so it adds no generation or reward-model cost.

  2. Consistent gains on two objectives. RAY-GRPO has the highest nHV on all 7 tasks against 12 baselines (Table 8), and the gains also hold on two other models (Table 12).

Critical weaknesses
  1. The three-objective claim is weaker than stated. GDPO has higher nHV on both transfer sets (Table 9). The higher Overall mostly comes from humor, which has a larger range than the other rewards (Table 7). Also, M = 128 was chosen on these same evaluations (Appendix C.1).

  2. All results use the training reward models and one training run. The same reward models are used for training and evaluation, so the gains may only come from fitting them. There is no LLM judge, human evaluation, or result over several seeds.

  3. The compute is not matched. Equal update counts do not mean equal rollouts or compute (Appendix C.1), and the paper says end-to-end cost comparisons are incomplete (Appendix B).

Questions for authors
  1. Given that GDPO has higher nHV on both three-objective sets, how should the three-objective result be read? Would the result hold if M were chosen on a separate split?
  2. Results over several training seeds on one task (e.g., HH2), and an LLM-judge check on a subset, would show whether the gains hold beyond the training reward models.
  3. How does RAY-GRPO compare with GDPO and Linear-GRPO at the same number of rollouts or the same GPU time?
  4. Does Linear-GRPO use the same endpoint/residual structure? If not, the gain may come from the structure rather than the ray-based weights. Results for Linear-GRPO with the same structure would make this clear.
  5. When the preference moves toward one objective, does that reward go up in a steady way? With τ = 0.02, helpfulness drops to almost zero (Table 11), which suggests the control can fail.
3 comments
AuthorsOpenReview ↗

Reply to gLNH (1/2)

We separate reward quality, frontier quality and preference control below.

Q1. Given GDPO’s higher nHV on both three-objective sets, how should these results be interpreted, and would they hold if ray count MM were selected on a separate split?

The K3 results show transfer of useful reward trade-offs without target-domain training. At the reported setting, RayGRPO improves Helpfulness as well as Humor over GDPO on both datasets, while GDPO retains higher Harmlessness. Thus, the gains extend beyond Humor, although raw Overall remains scale-dependent.

The table includes the full retained ray-count comparison. These are existing seed-0 evaluations with common prompts, controls and reporting geometry within each dataset. PSoups denotes Personalized-Soups. The former M=512 default retains its earlier training calibration.

DatasetSettingHelpfulness / Harmlessness / HumorOverallnHV
PSoupsRayGRPO, M=128-0.9038 / 1.1621 / 3.07771.11200.9022
PSoupsRayGRPO, M=512-0.8369 / 1.0590 / 2.71790.98000.9376
PSoupsRayGRPO, M=2048-0.8571 / 0.9729 / 2.85110.98900.9282
PSoupsGDPO-1.2687 / 1.2844 / 2.73590.91720.9170
HelpSteer2RayGRPO, M=128-0.0894 / 0.1493 / 2.20750.75580.8380
HelpSteer2RayGRPO, M=5120.1126 / 0.1417 / 1.60780.62070.8395
HelpSteer2RayGRPO, M=20480.0003 / -0.0358 / 1.85680.60710.8333
HelpSteer2GDPO-0.4088 / 0.2749 / 1.50380.45660.8413

Overall averages the evaluated reward vectors, whereas nHV measures the volume covered in reward space. GDPO's higher nHV at the reported setting is therefore compatible with RayGRPO's higher Overall. All retained ray-count settings exceed GDPO in Overall on both domains, so the observed advantage is not confined to the selected M=128. Selection of M used these evaluations; the requested independent-split test has not been conducted.

Q2. Do several training seeds on one task and an external LLM-judge subset support gains beyond the training reward models?

Yes. We provide complementary evidence from independent training seeds, blinded human judgments and an external LLM judge.

Independent training runs. We trained each method three times with different random seeds, using a common boundary/interior schedule, rank count and microbatching. All trained policies were evaluated on the same 512 prompts × 11 controls with greedy decoding. RayGRPO exceeds GDPO in nHV and Overall for each training seed; computational costs are reported in Q3.

Overall and nHV report mean ± SD across training runs.

MethodMean Helpfulness / HarmlessnessOverallnHVMean / worst steerability
RayGRPO1.0221 / 0.81520.9187 ± 0.03951.0558 ± 0.01930.9939 / 0.9879
GDPO1.3391 / 0.22220.7807 ± 0.04200.7901 ± 0.01660.9652 / 0.9485

Blinded human evaluation. Two raters compared final RayGRPO and GDPO policies at three controls, covering 135 distinct prompts and 270 main judgments. Model identities were masked, and all main judgments are retained. Both external evaluations score wins/ties/losses as 1/0.5/0. Their evaluation CIs use 10,000 resamples of retained prompts with replacement, keeping policies, answers and judgments fixed.

Human criterionResult (95% CI for preference scores)
Overall RayGRPO preference score60.2% [53.3,66.9]
Helpfulness59.6% [52.8,66.1]
Harmlessness59.8% [54.1,65.4]

The combined blinded ratings favor RayGRPO over GDPO in Overall preference, with Helpfulness and Harmlessness judgments pointing in the same direction. This provides human evidence that RayGRPO's advantage extends to perceived response quality in the evaluated HH2 sample.

External LLM evaluation. GPT-5.4 (gpt-5.4-2026-03-05) judged model-masked answers in both orders for 1,000 RayGRPO response samples spanning 428 distinct prompts and 11 controls. We average both answer orders with equal weighting across evaluation groups; bootstrap resampling groups all comparisons by original prompt. The same answer-completeness rule is applied to every method.

RayGRPO preference scores are reported with 95% pointwise confidence intervals.

ComparatorOverallHelpfulnessHarmlessness
GDPO58.1% [55.2,61.0]58.0% [55.1,60.9]53.5% [52.3,54.7]
Linear-GRPO63.0% [60.4,65.6]63.0% [60.4,65.6]53.0% [51.8,54.2]
HoE60.7% [57.9,63.7]60.7% [57.9,63.6]53.3% [52.1,54.6]
Rewarded Soups61.6% [58.6,64.7]61.6% [58.6,64.6]54.0% [52.6,55.4]

Across the evaluated responses, the judge favors RayGRPO over every baseline in Overall, Helpfulness and Harmlessness, with larger preference margins for Helpfulness and all pointwise intervals above indifference.

AuthorsOpenReview ↗

Reply to gLNH (2/2)

Q2 (continued).

Together with the blinded human preference for RayGRPO over GDPO, this supports a response-quality advantage beyond the training reward models.

Q3. How does RayGRPO compare with GDPO and Linear-GRPO at equal rollout counts or equal GPU time?

The historical GDPO run in the original HH2 comparison already matched RayGRPO in the number of prompt draws, rollouts per prompt and optimizer-update count. For RayGRPO, GDPO and Linear-GRPO, each original prompt produced two responses at each of four controls: a 1:8 prompt-to-rollout ratio. GDPO's four ranks × two prompts per rank and RayGRPO's eight ranks × one prompt per rank therefore both yield eight prompts and 64 responses per batch. With one PPO epoch per batch, each run used 40,016 prompt draws, 320,128 responses and 5,002 optimizer updates. GDPO's objective-wise normalization and batch whitening reuse these same responses. Its smaller microbatches accommodated memory constraints while preserving this sampling ratio and update count.

We retained these historical sampling and update budgets in the additional three-seed RayGRPO/GDPO experiment in Q2, while standardizing rank count, microbatching and boundary/interior scheduling. The corresponding accelerator costs, averaged over training seeds, are reported below.

MethodTraining accelerator-hoursEvaluation accelerator-hours
RayGRPO208.727.06
GDPO210.577.02

Q4. Does Linear-GRPO use the same endpoint/residual structure, and can this comparison distinguish structural gains from ray-based weighting?

Yes. The original HH2 Linear-GRPO baseline already uses the same endpoint/residual parameterization as RayGRPO: two rank-16 endpoint adapters, one for each objective, combined using the requested preference weights, plus a rank-16 interior residual that vanishes at the endpoints. Both adapt the attention q/k/v/o projections and contain 46.01M trainable adapter parameters.

Linear-GRPO forms advantages by linearly scalarizing rewards and standardizing them within each prompt/control group. The manuscript's HH2 Linear-GRPO results therefore already compare against RayGRPO using this shared adapter structure.

Q5. Does each reward increase steadily with its preference weight, and why does τ=0.02\tau=0.02 drive Helpfulness almost to zero in Table 11?

Monotonicity. Raw rewards increase approximately, but not strictly, monotonically with their own preference weight on the evaluated HH2 grid. For the three-seed study in Q2, we compute Spearman correlation ρ\rho between each objective's weight and its mean raw reward over 512 prompts. Adjacent decreases count violations among the ten successive weight increases:

Training seedHelpfulness / Harmlessness ρ\rhoHelpfulness decreasesHarmlessness decreases
00.9909 / 1.00001/100/10
11.0000 / 0.99090/101/10
20.9818 / 1.00002/100/10

These raw-reward measurements directly support consistent directional control. Strict monotonicity is not enforced: preferences guide ray allocation, while finite policy optimization does not impose an ordering constraint across controls.

Why did Helpfulness fall in Table 11? Table 11 reports Helpfulness averaged across controls. Rechecking the original default (transport temperature 0.05) and low-temperature (0.02) policies shows that the interior trade-offs shifted toward Harmlessness, while preference ordering largely survived. Late-training KL loss below averages logged interior updates at steps 4,000–5,001.

Original-run diagnosticDefaultLow temperature
Mean Helpfulness / Harmlessness1.2251 / 0.79470.0120 / 1.5020
Helpfulness / Harmlessness at equal weights2.0813 / 0.54920.1027 / 1.9596
Helpfulness at the helpful endpoint2.63552.2419
Helpfulness / Harmlessness ρ\rho1.0000 / 1.00000.9727 / 1.0000
nHV1.07861.0063
Late-training KL loss0.51800.7229

Lower transport temperature sharpens control–ray assignments, reducing averaging across directions. Because each ray credits its limiting reward, this changes the reward-dependent learning coefficients and can make training more sensitive to the evolving reward geometry. The original logs show higher late-training KL relative to the training reference despite converged transport and nonzero credit. Our interpretation is an optimization imbalance under sharper allocation: Helpfulness deteriorated at interior controls, rather than preference conditioning disappearing.

Thanks for the responses. The three-seed runs, the human and LLM-judge evaluations, and the matched compute address most of my concerns. On the three-objective sets, GDPO still has higher nHV on HelpSteer2 for every M, so I read that part as a trade-off rather than a clear gain.

Reviewer_eD9AOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces RAY-GRPO, a multi-objective alignment method that derives adaptive reward coefficients for Group Relative Policy Optimization (GRPO) from both the preference control ww and the policy's current reward attainment. Each control is mapped, via balanced entropic (Sinkhorn) transport, onto a distribution over MM reference rays drawn from a scrambled Sobol sequence on the positive unit sphere, where each ray uu specifies target reward proportions in a calibrated reward space. Each ray's utility is the exact radial attainment (min⁡kqk/uk)K\big(\min_k q_k / u_k\big)^K, with qq the clipped calibrated reward vector, and averaging these utilities over the transport plan yields a ray-smoothed attainment objective JJ that lower-bounds a finite-ray hypervolume estimator. Differentiating this average at cross-fitted calibrated reward means qˉ\bar{q} produces per-objective coefficients gsg_s that emphasize whichever objectives currently limit attainment. These coefficients weight leave-one-out group-relative reward contrasts into a single scalar GRPO advantage, reusing already-scored rollouts with no additional generations, reward-model calls, or per-objective gradient estimates. A single conditional policy is trained by alternating boundary updates (single-objective endpoint LoRA blocks at the simplex vertices eke_k) with interior refinement of a control-conditioned gated residual that vanishes at vertices, exactly preserving current endpoint behavior during refinement. The paper proves a finite-ray hypervolume lower bound for the surrogate and a nonzero-credit guarantee under strictly positive means qˉ≻0\bar{q} \succ 0, while explicitly scoping what these results do not imply. Experiments cover seven two-objective tasks against thirteen baselines from four methodological families, three-objective transfer to two held-out datasets, a five-objective HelpSteer comparison, HH2 ablations, and two additional backbones. RAY-GRPO achieves the highest normalized hypervolume on all seven K=2 tasks (10–14% over the strongest baseline, GDPO, on the assistant tasks), the highest Overall reward on both K=3 transfer sets (though GDPO attains slightly higher nHV there), and the best score on every K=5 objective. Ablations support the balanced allocation and the default (τ,M)(\tau, M), though the mechanism-removal ablation changes several components jointly and the K=3 ray count was selected on the evaluation data, which the authors flag as exploratory.

Critical strengths

S1. Adaptive credit mechanism. Differentiating the ray-averaged utility Eu[(min⁡kqk/uk)K]\mathbb{E}_u\big[\big(\min_k q_k/u_k\big)^K\big] at cross-fitted means qˉ\bar{q} yields coefficients gsg_s that concentrate on attainment bottlenecks and plug directly into GRPO's group-relative advantage, reusing scored rollouts with no extra generations, reward-model calls, or per-objective gradient estimates. This matters because it offers a theoretically grounded drop-in replacement for the fixed linear scalarization that is standard practice in multi-objective GRPO.

S2. Consistency. The evaluation spans thirteen baselines across four families, K$\in$ {2, 3, 5}, three backbones, and ablations; RAY-GRPO attains the highest nHV on all seven K=2 tasks with 10–14% margins over GDPO on the assistant tasks and wins every objective at K=5 on HelpSteer. This breadth substantially reduces the risk that the gains are an artifact of one task, comparator, or backbone.

S3. Reproducibility. The authors explicitly state what the theory does not imply (JJ-improvement does not entail HV-improvement; cross-fitting does not yield an unbiased gradient of the population JJ), flag exploratory choices (selecting the K=3 ray count on test data), admit imperfect baseline parity, and retain hashed prompts, raw reward vectors, and per-run configurations. This disclosure materially aids verification and raises confidence in the reported numbers.

Critical weaknesses

W1. Attribution of the gains is confounded, and the closest adaptive-weighting comparator is missing. The headline results bundle two contributions: ray-derived credit and boundary/interior adapter training and the single mechanism-removal ablation changes adapter structure, training schedule, credit rule, and KL reference simultaneously (as the authors admit), so it cannot isolate the source of the improvement. More importantly, the paper's core claim is that policy-dependent weighting beats static scalarization, yet dynamic/dual-variable reward weighting for multi-objective alignment (Lu et al., TACL 2026) is not included as a baseline. A factorial comparison plus a head-to-head against that method are needed to support the central claim.

W2. The guarantees are weaker than the abstract's framing. What is proven is that J(w,μ)J(w,\mu) lower-bounds a finite-ray hypervolume estimator and that λs,k>0\lambda_{s,k} > 0 when μ^k>0\hat\mu_k > 0 for all kk; neither yields policy improvement, hypervolume improvement, or Pareto-front quality, and the coefficients actually used come from clipped, RMS-normalized plug-in means rather than the population quantity in the lemma. The abstract's phrase "admits a finite-ray hypervolume lower bound and nonzero direct credit" invites a stronger reading than the theorems support, and the empirical nHV gains mitigate but do not close that gap.

W3. Statistical robustness and hyperparameter stability are not established. Results appear to come from one training run per configuration, with prompt-level bootstrap intervals conditional on that trained policy and reported marginally rather than as paired differences, so the small Reddit margins (0.2687 vs. 0.2659, 0.4342 vs. 0.4223, 0.4100 vs. 0.4034) do not clearly exceed plausible run-to-run noise. In addition, τ\tau = 0.02 collapses helpfulness to about 0.01, showing real sensitivity to a hyperparameter whose selection was partly informed by evaluation data at K=3.

Yining Lu, Zilong Wang, Shiyang Li, Xin Liu, Changlong Yu, Qingyu Yin, Zhan Shi, Zixuan Zhang, and Meng Jiang. Learning to optimize multi-objective alignment through dynamic reward weighting. Transactions of the Association for Computational Linguistics, 2026. doi:10.1162/TACL.a.696. arXiv:2509.11452.

Questions for authors

Q1. Can you provide paired bootstrap intervals (or per-prompt paired differences) for nHV and Overall against GDPO on each K=2 task, and report variance over at least 2–3 training seeds for the HH2 default? Without these, margins such as 0.2687 vs. 0.2659 on Reddit are difficult to interpret.

Q2. How should a practitioner map a desired exchange rate to a control ww? Since ww enters only as an angular prior via ℓ2\ell_2-normalized Euclidean distance to rays, with no monotone raw-reward guarantee, did you measure preference responsiveness (e.g., correlation between wkw_k and the attained reward share) across the evaluation grid?

Q3. How sensitive are the results to the choice of rref,kr_{\mathrm{ref},k} and the calibration quantiles, and what fraction of training steps trigger the numerical validity rejections (negative calibrated means, ∣gs∣<10−8|g_s| < 10^{-8})?

Q4. The advantage retains the full magnitude of gsg_s, which scales as ρK−1\rho^{K-1} with ρ=min⁡kqˉk/uk\rho = \min_k \bar{q}_k / u_k; did you compare against using only the normalized direction λs\lambda_s, and does coupling coefficient magnitude to attainment level affect early-training stability when ρ\rho is small?

Q5. Can you report end-to-end wall-clock/GPU cost against GDPO and Linear-GRPO at matched budgets (currently stated as incomplete), and clarify the equal-budget status of the K=3/K=5 MO-GRPO adaptations given the differing rank counts and reconstructed inference settings?

5 comments
AuthorsOpenReview ↗

Response to Reviewer eD9A — Part 1/4: Q1–Q2

We address your numbered questions first, followed by credit attribution, the additional DRW baseline and theory.

Q1. Can you provide paired nHV/Overall intervals against GDPO on every K=2 task and variance over at least 2–3 HH2 training seeds?

We trained RayGRPO and GDPO with three different random seeds each on HH2. GDPO is the strongest nHV baseline across all seven K=2 tasks. The other supplementary HH2 studies below use the same replication setup.

RayGRPO and GDPO share the adapter architecture, boundary/interior schedule, rank count, microbatching and sampling/update budgets described in Q5. Final policies are evaluated on identical 512 prompts × 11 controls with greedy decoding. RayGRPO exceeds GDPO in nHV and Overall in every seed. Overall averages raw rewards across objectives and controls; steerability is the Spearman correlation between each objective's weight and its raw reward.

Unless indicated otherwise, supplementary HH2 tables report means across training runs, with SDs where shown.

MethodMean Helpfulness / HarmlessnessOverallnHVMean / worst steerability
RayGRPO1.0221 / 0.81520.9187 ± 0.03951.0558 ± 0.01930.9939 / 0.9879
GDPO1.3391 / 0.22220.7807 ± 0.04200.7901 ± 0.01660.9652 / 0.9485

Single-run evaluation CIs. The original Table 8 results use one trained checkpoint per method. We resample prompts with replacement 2,000 times, keeping both methods and all eleven controls paired. Each replicate recomputes the front, Overall and nHV from saved outputs. The 2.5th and 97.5th percentiles of RayGRPO–GDPO differences give evaluation-prompt CIs; the SDs above summarize variability across training runs.

Original taskΔnHV evaluation CIΔOverall evaluation CI
Helpfulness / Harmlessness[0.1205,0.1487][0.3319,0.4017]
Helpfulness / Humor[0.1275,0.1687][0.0694,0.1875]
Harmlessness / Humor[0.1404,0.1858][0.4916,0.6099]
Summary / faithful[0.0027,0.0029][0.3632,0.3746]
Summary / DeBERTa[0.0132,0.0149][0.7715,0.8367]
Faithful / DeBERTa[0.0070,0.0085][1.7802,1.8494]
Reward / negative cost[0.0118,0.0185][2.9884,3.3876]

The paired intervals favor RayGRPO on every task. Pairing uses retained raw prompts and declared protocols; historical effective-input tokens were not retained. The reanalysis uses the available Table 8, whose Reddit entries differ from two values quoted in the review.

Q2. How should a desired exchange rate map to control ww, given its angular-prior role without a raw-reward monotonicity guarantee, and how do preference weights correlate with attained reward shares across the grid?

Choosing a control. We recommend sweeping controls on a separate validation set and recording the resulting mean raw rewards. For a desired exchange rate, score these operating points using the user's chosen raw-reward weights and select the best tested control. This gives an empirical mapping: the input control specifies a direction in calibrated reward space, rather than a fixed raw-reward exchange rate.

Does the control change the achieved trade-off? Following your suggested diagnostic, we measure how preference weights relate to attained reward shares:

s_k(w)=\frac{q_k(w)}{\sum_{j=1}^{K}q_j(w)}.$$ Here, $w$ is the input preference vector and $w_k$ its weight for objective $k$; $K$ is the number of objectives, and $j$ indexes all objectives in the sum. $\bar r_k(w)$ is the mean raw reward over evaluation prompts, $r_{\mathrm{ref},k}$ the fixed reference reward, and $D_k>0$ the fixed calibration width. Thus, $q_k(w)$ is normalized reward above the reference, and $s_k(w)$ is its share of the total across objectives. The table summarizes Helpfulness across the eleven evaluated controls. **Span** is the largest minus smallest attained share; **rank correlation** is Spearman correlation between its input weight and attained share. Entries list training seeds 0 / 1 / 2. | Method | Helpfulness-share span | Weight–share rank correlation | |---|---|---| | RayGRPO | 0.3972 / 0.3933 / 0.4019 | 1.0000 / 1.0000 / 1.0000 | | GDPO | 0.0994 / 0.1272 / 0.1267 | 1.0000 / 1.0000 / 1.0000 | Both methods produce ordered share responses on this grid, while RayGRPO covers a wider achieved trade-off range. This supports practical control selection; share ordering does not imply strictly monotonic raw rewards. ```
AuthorsOpenReview ↗

Reply to eD9A (2/4)

Q3. How sensitive are reference points and calibration quantiles, and what fraction of training steps trigger validity rejections for negative calibrated means or near-zero credit?

Reference and calibration sensitivity. Reference subtraction cancels in leave-one-out contrasts but affects the means forming credit coefficients; earlier K3 reference-margin tests showed sensitivity. The HH2 study below tests calibration and temperature while keeping the raw reference fixed at (−4, −5).

All settings share a new calibration sample, cached CPU transport (2,000 solver iterations), training budgets and greedy evaluation on 512 prompts × 11 controls under fixed reporting geometry. Calibration quantiles are 0.01/0.99 (default), 0.005/0.995 (wide) and 0.05/0.95 (narrow). The temperature arm changes only transport temperature from 0.05 to 0.02, addressing the sensitivity concern in W3.

SettingHelpfulness / HarmlessnessOverallnHVMean / worst steerability
Default1.1642 / 0.79020.9772 ± 0.05731.0703 ± 0.00980.9591 / 0.9182
Wide calibration0.7425 / 0.87850.8105 ± 0.17911.0368 ± 0.05740.9803 / 0.9697
Narrow calibration0.9250 / 0.92950.9273 ± 0.32451.0518 ± 0.05970.9818 / 0.9667
Lower temperature1.0916 / 0.82300.9573 ± 0.12821.0597 ± 0.02540.9773 / 0.9545

Preference ordering remains strong across all settings. The default has the highest observed mean Overall and nHV and the smallest SDs. Changing quantiles preserves ordering but increases reward variability.

Validity rejections. Your question specifically concerns negative reference-shifted calibrated means and near-zero credit vectors. We audited these conditions in all twelve completed training lineages:

Validity checks in completed lineagesRecorded count
Total / interior updates60,024 / 30,012
Checked fold fronts / calibrated coordinates / nonvertex covectors60,024 / 480,192 / 120,048
Reference-shifted coordinates below −10−8-10^{-8}0
Nonfinite front or covector events0
Nonvertex covectors with L2 norm ≤10−8\leq10^{-8}0

No monitored validity check failed in these completed lineages; an exact rejection fraction across all interrupted attempts is unavailable.

Q4. How does full credit magnitude compare with normalized direction, and does its ρK−1\rho^{K-1} scaling affect early-training stability when ρ\rho is small?

What the ablation changes. The credit vector gsg_s determines both relative objective weights and the strength assigned to a control. Each ray contributes a factor ρK−1\rho^{K-1}, where ρ\rho measures attainment along that ray from the reference. We compared full credit with λs=gs/∥gs∥1\lambda_s=g_s/\lVert g_s\rVert_1, preserving the relative objective weights but removing each vector's magnitude. We apply this separately to each interior control and held-out prompt fold, before the leave-one-out advantage calculation and the unchanged batch RMS normalization. A shared RMS scaling does not generally remove differences in strength between controls.

We compare two training designs. The proposed split design uses endpoint adapters plus an interior residual, alternating updates and an endpoint-based interior KL reference. The joint design uses one conditional adapter, joint updates and a fixed base-model reference. Within each design, architecture, training budgets, optimizer and endpoint credit stay fixed.

Design / creditMean Helpfulness / HarmlessnessOverallnHVMean / worst steerability
Split / full1.0221 / 0.81520.9187 ± 0.03951.0558 ± 0.01930.9939 / 0.9879
Split / direction only0.4187 / 1.09010.7544 ± 0.23350.9964 ± 0.05070.8076 / 0.7727
Joint / full1.2140 / 0.03770.6259 ± 0.00580.6583 ± 0.00090.7470 / 0.5879
Joint / direction only1.2528 / 0.03080.6418 ± 0.00370.6673 ± 0.00620.8212 / 0.7121

Full magnitude improves final nHV in every seed of the proposed split design and preserves stronger preference ordering. This supports retaining magnitude in our design. The joint design favors direction-only; there, normalization also changes interior credit relative to the fixed endpoint credit. The benefit therefore depends on the training design.

Early-training stability. For the split design, the table reports mean interior pre-clipping gradient norms and total clipped or skipped updates in the planned early windows. Entries are full / direction only.

Update windowMean gradient normClipped updatesSkipped updates
1–1000.0612 / 0.05930 / 00 / 0
101–5000.0636 / 0.06320 / 00 / 0

Both variants show similar early optimizer behavior. These windows were not conditioned on small ρ\rho, so they do not establish a distinct low-attainment stability advantage.

AuthorsOpenReview ↗

Part 3/4: Q4 continued; Q5; W1; DRW results

Q4 (continued).

The favorable evidence for full magnitude is its final frontier quality and preference control in the proposed design.

Q5. What are matched-budget wall-clock/accelerator costs versus GDPO and Linear-GRPO, and is K3/K5 MO-GRPO budget parity preserved despite different rank counts and reconstructed inference settings?

The original HH2 GDPO and Linear-GRPO runs already matched RayGRPO's 1:8 prompt-to-rollout ratio (four controls with two responses each), prompt exposure and optimizer-update count. With one PPO pass per batch, each used 40,016 prompt draws, 320,128 responses and 5,002 updates. The RayGRPO/GDPO study in Q1 retains those budgets and standardizes execution and boundary/interior scheduling. Equal accelerator-time limits would not ensure equal training exposure across execution configurations, so we report costs separately. Costs for the Q1 comparison are averaged over training runs:

MethodTraining wall hoursTraining accelerator-hoursEvaluation accelerator-hours
RayGRPO26.09208.727.06
GDPO26.32210.577.02

Historical HH2 RayGRPO, Linear-GRPO and GDPO use the same 46.01M adapter capacity. Training used eight ranks for RayGRPO and four for each baseline, with different microbatch settings. Higher-K methods match response/update budgets, with prompt exposure and capacity recorded below:

TaskRayGRPO / MO-GRPO responsesUpdatesRayGRPO / MO-GRPO prompt drawsAdapter parameters (M)Training accelerator-hours
K3480,192 / 480,1925,002 each40,016 / 60,02461.35 / 92.01219.90 / 179.35
K5188,960 / 188,9601,181 each9,448 / 23,62092.02 / 153.3558.87 / 82.95

These are complete-method comparisons: K3 evaluation layouts differ, and historical K5 effective-input parity is unverified. Timers cover retained training/evaluation, excluding setup, calibration and unrecorded failure/idle allocations.

W1. Can gains be attributed to ray credit separately from boundary/interior adapters, training schedule and KL reference?

We treated endpoint adapters, alternating updates, endpoint credit and the endpoint-based KL reference as one coordinated design; the original ablation tested it as a whole. To directly address credit attribution, we conducted a RayGRPO/Fixed credit GRPO × split/joint experiment, changing only the interior coefficient rule within each package while holding routing, schedule, KL-reference rule, REINFORCE Leave-One-Out (RLOO) baseline, root-mean-square normalization and budgets fixed. Here, Fixed credit GRPO replaces the interior ray-derived coefficients with w/2. The split-RayGRPO result is in Q1; the remaining cells are:

Package / creditMean Helpfulness / HarmlessnessOverallnHVMean / worst steerability
Split / Fixed credit GRPO0.3383 / 1.45940.89881.08110.7500 / 0.6788
Joint / RayGRPO1.2140 / 0.03770.62590.65830.7470 / 0.5879
Joint / Fixed credit GRPO1.1806 / 0.04910.61490.65860.7985 / 0.6818
Fixed credit GRPO diagnosticResult
Mean steerability, training seeds 0 / 1 / 20.9818 / 0.3591 / 0.9091
Training seed 1 worst steerability0.2182

Relative to RayGRPO's raw-reward steerability in Q1, Fixed credit GRPO results suggest greater vulnerability to partial preference-conditioning collapse in the split design. This concerns ordered control: Fixed credit GRPO has higher mean split nHV, while joint RayGRPO/Fixed credit GRPO nHV is nearly equal. Within-package contrasts identify the credit-rule intervention; cross-package contrasts compare coordinated designs with different active gradient budgets. Fixed credit GRPO isolates the interior coefficient rule and is distinct from conventional Linear-GRPO; GDPO uses one front fold versus RayGRPO's two.

Additional baseline (W1). How does Dynamic Reward Weighting (DRW) compare?

We added a Qwen3-8B adaptation of Lu et al.'s gradient-based Dynamic Reward Weighting (DRW), alongside a static-weight control. Both use 5,002 updates and 320,128 generated responses, matching the RayGRPO response/update budget. We evaluate final policies on the same 512 HH2 prompts and compare DRW with RayGRPO's predeclared equal-weight control. CIs estimate each method's mean Overall; training intervals use t with two degrees of freedom:

MethodMean Helpfulness / HarmlessnessOverall [evaluation-prompt 95% CI]Overall training-run 95% CI
RayGRPO, equal-weight control1.8419 / 0.71611.2790 [1.2292,1.3305][1.2376,1.3204]
DRW, final policy2.0689 / 0.32451.1967 [1.1391,1.2525][1.1783,1.2150]
Static-weight control2.0122 / 0.31651.1644 [1.1030,1.2216][1.1196,1.2091]

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal Experiments

Thank you for the constructive feedback. We completed the following supplementary studies and incorporated the results into Appendices J–M of the revised manuscript:

  1. Blinded human evaluation. Two raters provided 270 judgments on 135 distinct prompts. The combined results favor RayGRPO over GDPO in Overall, Helpfulness, and Harmlessness, supporting improvements in perceived response quality.

  2. External LLM evaluation. GPT-5.4 evaluated 1,000 RayGRPO response samples against GDPO, Linear-GRPO, HoE, and Rewarded Soups, with masked model identities and both answer orders. RayGRPO is preferred across all three criteria against every comparator; all reported pointwise confidence intervals exceed indifference.

  3. Controlled credit ablations. Holding architecture, schedule, KL reference, and training budgets fixed within each design clarifies the contribution of ray-derived credit. In the proposed boundary/interior design, RayGRPO maintains stronger preference ordering, while fixed credit is more vulnerable to partial preference-conditioning collapse. Retaining credit magnitude also improves frontier quality and preference ordering over direction-only credit in this design.

  4. Additional adaptive-weighting baseline. We implemented and evaluated an adaptation of Dynamic Reward Weighting (DRW). RayGRPO’s predeclared equal-weight control achieves higher Overall than DRW’s final policy across the evaluated training runs, while providing inference-time preference control. This comparison concerns final operating points.

  5. Calibration and temperature sensitivity. We completed twelve HH2 training runs: default calibration, wider calibration, narrower calibration, and lower transport temperature, each with three independent training seeds. Preference ordering remains strong across all four settings, while calibration changes affect reward quality and variability. Audits of the completed training lineages found no monitored validity violations.

  6. Three independent training seeds against GDPO on HH2. We trained RayGRPO and GDPO with three random seeds per method, using shared architecture, scheduling, execution settings, and sampling/update budgets. RayGRPO exceeds GDPO in both Overall and nHV in every training seed, with comparable measured training cost. This directly addresses reproducibility and training-budget concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.