- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
RAY-GRPO adapts the reward coefficients used by GRPO according to both the requested preference and the policy's current reward attainment. For each nonvertex control, balanced transport assigns a distribution of reference rays in calibrated reward space. The method measures attainment along each ray by its limiting reward-to-proportion ratio and differentiates the resulting utility to give more credit to objectives that currently limit the desired balance. It also alternates updates to vertex-specific LoRA blocks with updates to a control-conditioned residual for nonvertex trade-offs. The experiments cover multi-objective alignment; the clearest gains are on the seven two-objective tasks, while frontier-coverage results are more mixed in the other settings.
Critical strengths
- The reward-allocation rule has a clear interpretation: a ray is limited by the objective with the smallest attained-reward-to-required-proportion ratio. The derived coefficients reuse scores from existing rollouts, so the method does not require extra generations or reward-model evaluations.
- The boundary/interior parameterization is carefully designed. During nonvertex refinement, the endpoint blocks are frozen and the residual vanishes at simplex vertices, so those updates leave the current endpoint policies unchanged.
- The empirical study is broad. It includes thirteen methods on seven two-objective tasks, transfer evaluation with three objectives, a five-objective setting, ablations, and HH2 results on two additional backbones. The appendix is unusually explicit about configurations and comparison limitations.
Critical weaknesses
- Training and evaluation rely on the same frozen reward models. The paper itself appropriately describes the results as controlled reward-model measurements, but held-out evaluators or human preferences are needed before interpreting them as broader alignment improvements.
- The three-objective comparison is exploratory because the reported evaluations were also used to select the ray count. The five-objective experiment covers only three controls on a restricted simplex path, so it cannot establish dense preference coverage or responsiveness over the full simplex.
- Baseline parity is incomplete. Appendix C notes differences or missing evidence concerning truncation, prompt grouping, LoRA capacity, update and response budgets, GPU time, and historical inputs. These caveats make some of the cross-method gains difficult to attribute solely to RAY-GRPO.
- No variance across independent training runs is reported. In addition, the joint mechanism-removal ablation changes the adapter structure, endpoint credit, update schedule, and KL reference together, so it does not isolate which part of the boundary/interior design matters most.
Questions for authors
Please refer to the Weaknesses section.
4 comments
Reply to a3Lt (2/3)
Concern 2 (continued).
Steerability is the correlation between each objective's weight and its mean raw reward; the mean and worst values summarize the five objectives. Nondominated counts exclude control means dominated by another tested control; the single-box share measures how much total HV one control alone covers.
| Expanded K5 diagnostic | Result |
|---|---|
| Mean / worst steerability | 0.2867 / 0.2354 |
| Nondominated control means | 4 |
| HV accounted for by largest single control box | 99.9% |
The positive correlations show a modest control response beyond the original path, while the nondominated-count and volume-concentration diagnostics show that the frontier remains concentrated. We interpret K5 as evidence of achievable operating points, with preference responsiveness assessed separately from frontier volume. The expanded evaluation uses its own protocol and has no baseline comparison on this grid.
Concern 3. Are baselines comparable in input truncation, prompt grouping, LoRA capacity, generated-response/update budgets, GPU time and historical effective inputs?
On-policy HH2 comparisons. The historical GDPO and Linear-GRPO runs already matched RayGRPO's sampling and update budgets. Each original prompt produces two responses at each of four controls; eight prompt draws therefore yield 64 responses per optimizer update. With one PPO pass per batch, each method uses 40,016 prompt draws, 320,128 generated responses and 5,002 updates. All three have 46.01M adapter parameters. GDPO's normalization reuses the same responses; different rank counts and microbatch sizes preserve these totals. Thus, the original comparison matches prompt and generated-response exposure as well as the update-to-sample ratio.
The HH2 MO-GRPO baseline also matches these prompt, response and update totals, but samples eight responses at one control per prompt. For K3 and K5, MO-GRPO matches RayGRPO's generated-response/update budgets: respectively 96 responses per update for 5,002 updates and 160 for 1,181 updates. Its grouping gives it more prompt draws, and its adapter capacity is larger. These budgets preserve generated-response exposure and update frequency across different parameterizations.
Expert-based and additional baselines. Rewarded Soups trains two objective-specific experts, each with 10,005 updates and 16 responses per update, totaling 320,160 responses across both experts. This closely matches HH2's generated-response budget, but its optimizer-update count differs. HoE reuses those experts and adds router training and selection; these are additional costs. Other method-specific comparisons retain their documented training recipes: for example, the additional K3 MORLHF/PPO baseline uses its step-1,200 checkpoint with 9,600 responses. We do not describe these different training procedures as sharing one update-to-sample ratio.
Direct RayGRPO–GDPO comparison. We retained the historical sampling and update budgets in the additional three-seed HH2 study while standardizing the boundary/interior schedule, rank count and microbatching. The final evaluation also uses identical effective input tokens, controls and greedy decoding. Equal accelerator-time limits would not ensure equal training exposure across execution configurations, so we report measured training and evaluation costs separately. The following costs average the three seeds; performance and attribution results follow in Concern 4.
| Cost metric | RayGRPO | GDPO |
|---|---|---|
| Mean training accelerator-hours | 208.72 | 210.57 |
| Mean evaluation accelerator-hours | 7.06 | 7.02 |
These costs cover retained training and evaluation. The standardized HH2 study establishes input parity for this comparison; historical K3/K5 effective-input parity is not inferred from matching prompt lists alone.
Concern 4. Can independent training runs quantify variance, and which parts of the boundary/interior design matter when the ablation changes adapters, endpoint credit, update schedule and KL reference together?
Three training runs per method. We trained RayGRPO and our main comparator, GDPO, separately with three different random seeds each. The study retains the budgets in Concern 3 and standardizes architecture, boundary/interior schedule, rank count and microbatching. All trained policies were evaluated on the same 512 prompts × 11 controls with greedy decoding. RayGRPO exceeds GDPO in nHV and Overall for each training seed.
Overall averages raw rewards across objectives and controls; steerability is the objective-wise Spearman correlation between preference weight and raw reward. Overall and nHV report mean ± SD across training runs; the remaining columns average the training runs, with worst steerability taking the lower objective correlation within each seed before averaging.
Reply to a3Lt (3/3)
Concern 4 (continued).
| Method | Mean Helpfulness / Harmlessness | Overall | nHV | Mean / worst steerability |
|---|---|---|---|---|
| RayGRPO | 1.0221 / 0.8152 | 0.9187 ± 0.0395 | 1.0558 ± 0.0193 | 0.9939 / 0.9879 |
| GDPO | 1.3391 / 0.2222 | 0.7807 ± 0.0420 | 0.7901 ± 0.0166 | 0.9652 / 0.9485 |
Isolating interior credit. We also ran a two-by-two comparison of credit rule and training design, with three separate training runs in every cell. The split design above uses separate endpoint adapters and an interior residual, alternating boundary/interior updates, and an endpoint-based interior KL reference. The joint design uses one preference-conditioned adapter, joint updates, and a fixed base-model KL reference. Both designs retain pure-objective endpoint credit.
Within each design, Fixed credit GRPO replaces only the interior ray-derived coefficients with w/2, where w is the requested preference vector. Architecture, schedule, KL-reference rule, leave-one-out reward baseline, root-mean-square normalization and generated-response/update budgets are held fixed. The table uses the same summary definitions as above:
| Design / credit | Mean Helpfulness / Harmlessness | Overall | nHV | Mean / worst steerability |
|---|---|---|---|---|
| Split / RayGRPO | 1.0221 / 0.8152 | 0.9187 ± 0.0395 | 1.0558 ± 0.0193 | 0.9939 / 0.9879 |
| Split / Fixed credit GRPO | 0.3383 / 1.4594 | 0.8988 ± 0.4787 | 1.0811 ± 0.0863 | 0.7500 / 0.6788 |
| Joint / RayGRPO | 1.2140 / 0.0377 | 0.6259 ± 0.0058 | 0.6583 ± 0.0009 | 0.7470 / 0.5879 |
| Joint / Fixed credit GRPO | 1.1806 / 0.0491 | 0.6149 ± 0.0028 | 0.6586 ± 0.0038 | 0.7985 / 0.6818 |
To make conditioning reliability visible, the next table reports each split-design seed separately. Each entry gives mean / worst objective-wise steerability:
| Training seed | RayGRPO | Fixed credit GRPO |
|---|---|---|
| 0 | 0.9955 / 0.9909 | 0.9818 / 0.9727 |
| 1 | 0.9955 / 0.9909 | 0.3591 / 0.2182 |
| 2 | 0.9909 / 0.9818 | 0.9091 / 0.8455 |
Fixed credit GRPO is more vulnerable to partial preference-conditioning collapse in the observed split-design runs, while RayGRPO retains near-perfect ordering in every seed. Fixed credit GRPO's higher mean split nHV shows why frontier volume and reliable preference control should be assessed separately. The joint credit contrast has nearly equal nHV. Thus, the evidence supports ray credit's role in reliable control within the proposed design; within-design contrasts isolate the coefficient rule, while comparisons across designs assess the coordinated architecture, schedule and KL choices.
Dear Reviewer a3Lt,
Thank you again for your careful review. We have provided additional experiments and clarifications in response to your comments, and would appreciate your thoughts when you have a chance to revisit them. Please let us know if any concerns remain or if further explanation would be helpful. We would be glad to continue the discussion.
Thank you for your time and consideration.
Reply to a3Lt (1/3)
We organize the response around the four concerns in your Weaknesses section.
Concern 1. Given the same frozen reward models for training and evaluation, do held-out evaluators or human preferences support broader alignment claims?
Yes. Two raters compared final RayGRPO and GDPO policies across three controls with model identities masked, covering 135 distinct HH2 prompts and 270 main judgments. All main judgments are retained. Wins/ties/losses score 1/0.5/0. Evaluation CIs use 10,000 resamples of retained prompts with replacement, keeping policies, answers and judgments fixed. Combined results are:
The combined blinded ratings favor RayGRPO over GDPO in Overall preference, with Helpfulness and Harmlessness judgments pointing in the same direction. This provides human evidence that RayGRPO's advantage extends to perceived response quality in the evaluated HH2 sample.
GPT-5.4 (
gpt-5.4-2026-03-05) also assessed model-masked answers in both orders for 1,000 RayGRPO response samples spanning 428 distinct prompts and 11 controls. The evaluation retains the earlier 90 samples and judgments; 15 historical pairs were excluded for uncertain input matching. Earlier judgments preceded expansion and retain a separate analysis target. The same answer-completeness rule is applied to every method.The comparison includes 1,000 GDPO pairs and 995 pairs for each other baseline. HoE and Soups each have 994 complete pairs for Overall and Helpfulness, and 995 for Harmlessness. Scores use the same win/tie/loss rule, average both answer orders, and weight evaluation groups equally. Evaluation CIs use 10,000 resamples of retained prompts with replacement, keeping each prompt's comparisons together and policies, answers and judgments fixed.
RayGRPO preference scores are reported with 95% pointwise confidence intervals.
All lower bounds remain above indifference when missing orientations count as losses.
Across the evaluated responses, the judge favors RayGRPO over every baseline in Overall, Helpfulness and Harmlessness, with larger preference margins for Helpfulness and all pointwise intervals above indifference. Together with the blinded human preference for RayGRPO over GDPO, this supports a response-quality advantage beyond the training reward models.
Concern 2. Does selecting K3 ray count on the reported evaluations bias the results, and can K5’s three-control restricted path establish full-simplex coverage or responsiveness?
K3: transfer and ray-count selection. RayGRPO transfers useful reward trade-offs to Personalized-Soups (PSoups) and HelpSteer2 without training on either target domain. At the reported M=128 setting, it improves both Helpfulness and Humor over GDPO on both datasets, so the gain is not solely a consequence of Humor's scale. The full retained ray-count comparison is:
All retained settings exceed GDPO in Overall on both domains, including the former default M=512, which retains its earlier calibration. The advantage therefore persists beyond the selected setting. These transfer evaluations informed the choice of M; they do not constitute independent selection and testing. Overall averages raw rewards and remains scale-dependent, whereas nHV measures frontier volume: GDPO's higher nHV at the reported setting is compatible with RayGRPO's better Helpfulness and Humor.
K5: control beyond the reported path. We analyzed an existing expanded evaluation of the submitted RayGRPO K5 checkpoint on 43 controls × 503 prompts per control. It covers simplex vertices, equal-weight faces and asymmetric interior weights, extending beyond the manuscript's three path controls.