RAY-GRPO: MULTI-OBJECTIVE ALIGNMENT THROUGH RAY-GUIDED REWARD WEIGHTING

Group Relative Policy Optimization (GRPO) learns from within-group reward comparisons. Multiple objectives require reward credit that adapts to the policy’s changing bottlenecks. We introduce RAY-GRPO, with two components: raybased preference smoothing and policy-dependent reward reweighting. The first balances reference directions across controls and averages exact radial utilities, assessing several reward proportions at the same attained rewards. The second differentiates this average, giving each objective credit through the directions it currently limits. Auxiliary rays reuse observed rewards. Cross-fitted coefficients and alternating updates to endpoint low-rank adapters and a gated residual train one conditional policy while preserving endpoint behavior at vertices. The associated surrogate admits a finite-ray hypervolume lower bound and nonzero direct credit for strictly positive control means. Experiments with two, three, and five objectives demonstrate gains in overall reward and empirical Pareto-front quality against the evaluated baselines.