acceptodds
Under review as a conference paper at ICLR 2027

Don't Let Conflicting Rewards Pull Generative Recommendations Off Target

Abstract

LLM-based generative recommender systems must satisfy multiple objectives, but in practice these objectives may have different priorities, with recommendation accuracy serving as the primary objective and auxiliary objectives such as diversity and rationale coherence providing complementary training signals. In widely adopted reinforcement learning with verifiable rewards (RLVR) frameworks such as Group Relative Policy Optimization (GRPO), these heterogeneous rewards are merged into a fixed weighted sum, implicitly treating all rewards as equally tradable. When rewards conflict and their signals differ in scale, difficulty, and saturation, fixed aggregation can cancel gradients or let fast-improving auxiliary rewards dominate, resulting in reward hacking and signal collapse that can even degrade recommendation accuracy. To address this, building on the RLVR-style optimization framework, we propose Groupwise Gradient-Aligned Reward-Decoupled Policy Optimization (RPO), which dynamically selects reward-aggregation weights to maximize alignment with the primary accuracy gradient while constraining the resulting update to avoid substantial first-order degradation of auxiliary objectives. This yields a primary-centered update within the region defined by auxiliary compatibility constraints. Concretely, RPO computes per-reward aggregation weights via a simplex linear program, which identifies a weighted combination of the primary (i.e., accuracy) and auxiliary reward (i.e., diversity) gradients that is maximally aligned with the primary reward gradient, while enforcing that auxiliary gradients are non-adversarial to the resulting update direction. Experiments on five Amazon domains and MovieLens show generally consistent gains in accuracy, diversity, and rationale coherence. Beyond offline evaluation, our method is deployed in production and achieves the highest click-through rate in a real-world online A/B test.

Primary area:
generative models
Keywords:
Generative Recommendation, Reward Conflict

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_8EFqOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper tackles multi-reward RLVR for generative recommendation. Accuracy is treated as the primary reward; diversity and rationale groundedness serve as auxiliaries. The core idea is to normalize each reward separately, then solve a simplex-constrained LP that keeps the update aligned with the accuracy gradient while capping first-order interference from the auxiliary gradients. A format-compliance gate filters training samples. Results span five Amazon domains, MovieLens, a rationale-quality evaluation, and an online A/B test.

Critical strengths
  1. The gradient conflict analysis is convincing: accuracy and diversity gradients frequently oppose each other, and Figure 2 shows concrete cases where GRPO and GDPO actually hurt accuracy compared to SFT. This motivates the accuracy-first design well.
  2. The experimental coverage is broad — conventional and generative recommenders, alternative gradient-combination baselines, ablations, rationale quality, plus a production deployment. The weighting module's overhead looks small in the tested setting.
Critical weaknesses
  1. I cannot tell how much the LP itself adds over simpler alternatives. Figure 6 compares accuracy-only training with the full method, but only on two domains and without tabulated numbers; the main table also misses a matched SFT starting point. The optimization setup also needs clarification: the tolerance operates on scale-dependent gradient inner products, yet the paper says ε = 0.05 while the example config uses 0.0, and fallback statistics from the implementation are not reported. Without these details, it is hard to assess both the LP's marginal benefit and its actual behavior during training.

  2. Several choices in the online experiment are left unexplained. The deployed variant's modifications relative to the offline method are not detailed, the CTR difference lacks a user-clustered confidence interval, and the latency gap (60 ms vs. 138 ms) is reported without confirming matched hardware. Taken together, these gaps weaken the production evidence.

Formatting. The spacing around headings and figures looks artificially compressed, especially pp. 8–9; the template margins may have been modified. Small tables and tightly wrapped text on those pages are hard to read.

Questions for authors
  1. Given that the paper states ε = 0.05 but the example config uses 0.0, which value was actually used in the reported experiments? How sensitive are the weights and the accuracy/auxiliary trade-off to this tolerance?
  2. What do the fallback logs show? How often does constraint relaxation or primary-only fallback kick in, and how frequently is the primary gradient effectively zero?
  3. What was changed for the deployed variant, and which parts of the offline method were kept?
  4. Can you estimate uncertainty for the CTR difference using users as the clustering unit while keeping the impression-weighted CTR definition?
  5. Were the 60 ms and 138 ms latencies measured under the same hardware and serving conditions? What explains the gap, including any changes specific to the deployed variant?
  6. Could you add the matched SFT baseline to the main table, and extend the Figure 6 comparison to more domains with tabulated numbers?
4 comments
AuthorsOpenReview ↗

Response to Reviewer 8EFq (1/2)

We thank the reviewer for the careful review and the concrete questions. [G1]–[G3] refer to the Global Response.


[Q1] Value of ε, Config, and Sensitivity


Takeaway. All reported G²RPO results outside the ε sweep use ε = 0.05, the 0.0 in the example config was not used for any reported run, and 0.05 gives the highest HR@1 of the four values in all six domains.

Please refer to [G1] (1)–(3) in the Global Response.

  • All reported G²RPO results except the ε sweep in [G1] (2) use ε = 0.05 (Table 6). The 0.0 in the released example config is a default value that was not used for any reported run.
  • The value 0.05 gives the highest HR@1 of the four values tested in all six domains, and a larger ε reduces how often the fallback is used (4.7% → 0.2%); the full table is in [G1] (2).
  • On the auxiliary side, Div@1 at ε = 0.05 stays within 0.012 of the best value among the four settings in every domain ([G1] (2)).

The reviewer is right that the tolerance acts on scale-dependent raw inner products. The median auxiliary-to-accuracy gradient-norm ratios are 0.62–0.81, and cosine-normalizing the Gram matrix changes HR@1 by at most 0.12 ([G1] (3)).


[Q2] Fallback Logs and Zero Primary Gradient


Takeaway. Constraint relaxation and the primary-only fallback combined occur in 0.6% of steps at ε = 0.05, and the accuracy gradient g_p is zero in 0.4–4.3% of mini-batches.

Please refer to [G1] (4) in the Global Response.

  • Constraint relaxation and the primary-only fallback (w⋆ = e_p) are logged together: when the solver returns no solution, the auxiliary constraints are dropped one at a time, and e_p is used if this also fails. Combined, they occur in 0.6% of steps at ε = 0.05.
  • Separately, the accuracy gradient g_p, which is aggregated over the B = 4 groups of a mini-batch, is zero in 0.4–4.3% of mini-batches (per-domain values in [G1] (4)).

[Q3] Deployed Variant


Takeaway. The deployed model keeps the same G²RPO training procedure but uses production interaction data, retrieval-supplied candidates, and a business-rule post-filter applied identically to all models.

Please refer to [G3] (2) in the Global Response.

  • Kept: the deployed model is trained with the same G²RPO procedure as the offline models.
  • Changed (data): it is trained on the interaction data of the production service instead of the public datasets.
  • Changed (candidates): the candidate list in the prompt is supplied by the production retrieval stage (top-N) instead of the offline 200-item sampled set.
  • Changed (post-filter): outputs pass a business-rule post-filter that is applied identically to all models in the test.

[Q4] User-Clustered Uncertainty


Takeaway. Yes: with users as the clustering unit, the +0.16pp CTR gain over ML has a 95% CI of [+0.045, +0.271] (p = 0.008).

Please refer to [G3] (1) in the Global Response.

With users as the clustering unit (10,000 bootstrap resamples) and the impression-weighted CTR definition unchanged:

User-clustered bootstrap (10K resamples).

ComparisonΔCTR95% CI (user-clustered bootstrap, 10K)p
vs ML+0.16pp[+0.045, +0.271]0.008

[Q5] Latency


Takeaway. Yes: the 60 ms and 138 ms latencies were measured on the same hardware, and the latency gap is not caused by the LP.

Please refer to [G3] (3) in the Global Response.

  • The 60 ms and 138 ms latencies were measured on the same hardware under the same serving conditions.
  • The LP is solved only during training to set the reward weights; it requires no additional computation at inference, so the latency gap is not caused by the LP.
  • The gap comes from the larger model size and from generating the recommendation rationale. The changes specific to the deployed variant (retrieval-supplied candidates and the business-rule post-filter) have little effect on latency.
  • 138 ms is within our internal 150 ms latency budget.

[Q6] Matched SFT Baseline and Extended Fig. 6


Takeaway. Row (B) +SFT of Table 3 is the matched SFT starting point, and Table 3 replaces the Fig. 6 comparison with all six domains.

Please refer to [G2] (1)–(2) in the Global Response.

  • Row (B) +SFT of Table 3, which is in the main text, gives the matched SFT starting point; it is the common initialization of all RL-based variants and baselines.
  • The Fig. 6 comparison of the submission is replaced by Table 3, which covers all six domains with tabulated HR@1 and Div@1 for every variant, and the LLM-judge score where available.
AuthorsOpenReview ↗

Response to Reviewer 8EFq (2/2)

[W: Marginal Benefit of the LP]


Takeaway. On the identical pipeline, G²RPO is above PCGrad, CAGrad, and MGDA on every Table 1 entry except three, and the full method is higher than accuracy-only (E) on both HR@1 and Div@1 in all six domains.

(1) Simpler Alternatives on the Identical Pipeline.

PCGrad, CAGrad, and MGDA are run on exactly the same training pipeline as G²RPO: the same K = 3 per-reward gradient proxies from group-normalized, format-gated advantages, with only the LP weighting replaced by their update rules (Appendix C).

In Table 1 (HR@1 / HR@5 / NDCG@5 on six domains), G²RPO is above all three on every entry except:

  • Clothing HR@1 (PCGrad 10.90 vs 10.46);
  • CDs HR@5 (MGDA 37.63 vs 36.54);
  • MovieLens HR@1 (MGDA 3.75 vs 3.29).

(2) Accuracy-Only and Fixed Weights vs Full.

The most direct tests of the LP compare the full method (I) with accuracy-only (E), where w is fixed to e_p, and with fixed weights (H), where (w_acc, w_div, w_gr) = (0.84, 0.15, 0.01) for every mini-batch (variant letters as in Table 3 of the revised paper).

Excerpt of Table 3 (HR@1 / Div@1).

VariantFashionClothingGroceryHealthCDsMovieLens
(E) accuracy-only15.01 / 0.296810.11 / 0.295611.32 / 0.28725.11 / 0.097815.98 / 0.22292.40 / 0.0385
(H) fixed weights19.77 / 0.29028.98 / 0.277710.45 / 0.27997.14 / 0.113914.44 / 0.23972.76 / 0.0349
(I) full (per-batch LP)22.21 / 0.311510.46 / 0.296412.12 / 0.32887.14 / 0.118817.58 / 0.23933.29 / 0.0395

Each cell reports HR@1 / Div@1.

  • vs (E): the full method is higher on both HR@1 and Div@1 in all six domains.
  • vs (H): it has higher HR@1 in five domains and the same HR@1 in Health, and higher Div@1 in five domains (CDs: 0.2393 vs 0.2397).

The tight spacing still makes parts of the revised paper hard to read. I’ll leave template compliance to the AC and base my rating on the technical results.

I appreciate the additional work. The fixed-weight comparison and the user-clustered CTR analysis address my main concerns, so I’m raising my rating from 2 to 3.

Thank you so much for your follow-up and for raising your score! We will make sure to refine the spacing and readability in the final version.

Reviewer_3QnoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies multi-reward reinforcement learning for generative recommendation, prioritizing accuracy while considering diversity and rationale groundedness. It proposes G²RPO, which dynamically weights separately normalized rewards according to gradient alignment. Experiments on Amazon, MovieLens, and a production A/B test report improvements in accuracy, long-tail exposure, rationale quality, and click-through rate.

Critical strengths

S1: A practically relevant problem and a well-motivated method

The paper addresses an important problem in multi-reward training for generative recommendation: recommendation accuracy, diversity, and rationale quality may conflict, while accuracy is typically the primary objective in practical systems. Rather than treating all objectives equally, the proposed method prioritizes recommendation accuracy while limiting degradation in auxiliary objectives. This asymmetric design is well motivated and more appropriate for real-world recommendation settings than simple fixed-weight reward aggregation.

In addition, treating format compliance as a hard constraint rather than an ordinary reward is a sensible design choice, as it reduces the risk of the model receiving positive training signals for outputs that are correctly formatted but semantically incorrect.

S2: Comprehensive evaluation with real-world validation

The paper evaluates the proposed method across five Amazon domains and MovieLens, and compares it with collaborative filtering methods, LLM-based generative recommenders, and multi-objective optimization approaches. The results indicate that the method can achieve strong recommendation accuracy while also improving long-tail exposure and rationale quality across multiple settings.

The paper further reports an online A/B test in a production recommendation system. Although stronger controlled online baselines would improve attribution, the production deployment and statistically significant CTR improvement provide meaningful evidence of the method’s practical relevance.

Critical weaknesses

W1: Insufficient coverage of existing RL/LLM post-training methods and unclear positioning

The paper focuses primarily on GRPO, GDPO, and conventional multi-task gradient optimization methods, while providing limited discussion of the broader literature on LLM post-training. For example, methods such as On-Policy Distillation can provide denser token-level supervision through a teacher model, thereby addressing sparse rewards and credit assignment from a different perspective. Other policy optimization and constrained optimization methods may also improve training stability.

I recommend expanding the related-work discussion and including either a discussion or an empirical comparison with representative recent LLM post-training methods. If some of these methods are unsuitable as direct baselines because they require a teacher model or address a different problem setting, the paper should clearly explain these distinctions.

W2: Insufficient depth in the experimental analysis

The paper’s central claim is that dynamically adjusting the weights of different rewards can mitigate reward conflicts and prevent auxiliary objectives from pulling the model away from recommendation accuracy. However, the current experiments mainly demonstrate the method’s effectiveness through final performance metrics, without sufficiently analyzing the key phenomena during training.

Most importantly, the paper should provide a more detailed analysis of how the weights assigned to different rewards evolve throughout training and what their final distributions or values look like at convergence. Such analysis is necessary to verify that the proposed dynamic weighting mechanism behaves as intended and to clarify how it resolves conflicts among the reward objectives in practice.

Questions for authors

Q1: Could the authors provide a detailed analysis of how the reward weights evolve during training, including their final distributions across datasets and random seeds, and explain whether these dynamics consistently support the claim that G²RPO mitigates reward conflicts while preserving recommendation accuracy?

Q2: GRPO and GDPO do not appear to constitute the core contribution of the paper, yet they occupy a substantial portion of the manuscript. The necessity and extent of this discussion should be reconsidered and more clearly justified.

1 comment
AuthorsOpenReview ↗

Response to Reviewer 3Qno

We thank the reviewer for the thoughtful review and the constructive suggestions.

[W1] Positioning Relative to On-Policy Distillation


Takeaway. On-policy distillation uses token-level feedback from a teacher model, whereas our setting has no separate teacher; the two are complementary, and the revised paper adds this distinction (Section 2).

On-policy distillation (Agarwal et al., 2024) trains a student on its own generated sequences with token-level feedback from a teacher model, which densifies supervision. It has recently been applied to generative recommendation as well (OneRec Team, 2026).

Our setting has no separate teacher model: G²RPO is a multi-reward RLVR framework in which the accuracy, diversity, and groundedness rewards are computed for each rollout, and it addresses how the gradients of conflicting rewards are aggregated into a single update.

The two are therefore complementary rather than competing, and on-policy distillation is not a direct baseline in our teacher-free setting.

In the revised paper, we add a related-work discussion of on-policy distillation that states this distinction (Section 2, "Relation to On-Policy Distillation").


[W2 / Q1] Reward-Weight Dynamics


Takeaway. The LP weights keep most of the mass on accuracy, vary from batch to batch without drifting during training, and are nearly identical across seeds.

(1) Evolution During Training.

Fig. 4 shows the per-step evolution of the reward aggregation weights during training.

  • Unconstrained trainable weights concentrate around 0.3–0.4 across rewards, whereas the LP weights of G²RPO keep most of the mass on accuracy and vary from batch to batch.
  • The weights do not drift during training: for each of the three weights, the average over the first half and over the second half of training differ by at most 0.005 in every seed.
  • The realized parameter update stays accuracy-first, as intended: its first-order accuracy progress is positive in 95.1–97.0% of training steps, and its mean cosine similarity with the accuracy gradient is 0.66–0.77 (Appendix E, Table 8).

(2) Final Weight Distributions across Seeds.

The LP weights are computed for each mini-batch and logged per training run rather than per dataset, so we report their statistics across three random seeds.

LP weights across three seeds.

w_accw_divw_gr
Mean ± std across seeds0.836 ± 0.0070.148 ± 0.0020.013 ± 0.001
Last 10% of training, mean ± std across seeds0.840 ± 0.0120.146 ± 0.0140.011 ± 0.001

The average weights are nearly identical across seeds (std across seeds at most 0.007), and the weights at the end of training match the run average.

In the revised paper, the across-seed averages and the first-half vs second-half comparison are reported in Section 3.4, next to Fig. 4.

(3) Do the Dynamics Support the Claim?

Whether the per-batch variation of the weights matters, beyond their average, is tested against a baseline with fixed weights (w_acc, w_div, w_gr) = (0.84, 0.15, 0.01).

Fixed weights vs per-batch LP (HR@1 / Div@1).

MethodFashionClothingGroceryHealthCDsMovieLens
Fixed weights19.77 / 0.29028.98 / 0.277710.45 / 0.27997.14 / 0.113914.44 / 0.23972.76 / 0.0349
G²RPO (per-batch LP)22.21 / 0.311510.46 / 0.296412.12 / 0.32887.14 / 0.118817.58 / 0.23933.29 / 0.0395

Each cell reports HR@1 / Div@1.

G²RPO has higher HR@1 in five domains and the same HR@1 in Health, and higher Div@1 in five domains, so its gains are not explained by a fixed emphasis on accuracy alone.


[Q2] Length of the GRPO/GDPO Background


Takeaway. We condensed the GRPO/GDPO background in Section 2, keeping GDPO's per-reward group normalization (Eqs. 4–5) because G²RPO is built directly on it.

We condensed the GRPO/GDPO background in Section 2 of the revised paper.

We keep GDPO's per-reward group normalization (Eqs. 4–5) because G²RPO is built directly on it: the per-reward advantages of Eq. 4 are the inputs to our gradient proxies g_k and the LP, and the LP replaces GDPO's fixed uniform aggregation (Eq. 5).

Reviewer_8yMZOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes G²RPO for multi-reward RL post-training of generative recommenders. Recommendation accuracy is the primary objective, diversity and rationale coherence (groundedness) are auxiliary objectives. Building on per-reward group-normalized advantages, the method computes reward-gradient proxies over LoRA parameters and solves a simplex linear program to maximize alignment with the accuracy gradient while constraining negative auxiliary proxy projections. Experiments cover five Amazon domains and MovieLens show that the method achieves the highest value on 15 of 18 accuracy metrics and online CTR is 7.32%, compared with 7.16% for the strongest ML baseline.

Critical strengths

S1. Clear and practically relevant motivation. The authors explicitly state that “accuracy defines the optimization objective, while auxiliary rewards define compatibility constraints on the update direction.” Equation (14) operationalizes this distinction, while Figures 1–2 provide motivational evidence of gradient conflicts and auxiliary improvements accompanied by accuracy degradation. The conditional theoretical analysis is also useful: Theorem 1 correctly derives bounds for the specified update under its assumptions, and Theorem 2 provides a correct dual and complementary-slackness interpretation. S2. Relevant offline comparisons with important controls. Table 1 reports averages over three seeds and paired user-level significance tests. Besides recommendation and GRPO/GDPO-style baselines, the paper includes PCGrad, CAGrad, and MGDA. S3. Human validation of rationale evaluation and practical deployment evidence. Appendix F reports approximately 250 samples annotated by two experts. Rationale evaluation therefore does not rely solely on unvalidated LLM judgments. The online experiment reports impressions, clicks, user-sticky assignment, supporting the arithmetic consistency of results.

Critical weaknesses

W1. The conditional theoretical guarantee is not sufficiently connected to the actual training update. The proxy-gradient error δₖ is not quantitatively characterized, leaving the practical strength of the bound unclear. Moreover, advantage post-processing and AdamW may alter the LP direction, so the guarantee does not automatically extend to the implemented update. This concerns the applicability of the guarantee rather than the validity of the theorem under its assumptions. (Appendix E, Assumption 2 and Theorem 1, Algorithm 1, Table 5) W2. The incremental value of dynamic weighting requires more direct evidence. The paper includes uniform weighting D, accuracy-only weighting E, and trainable weighting G, but does not report a validation-selected non-uniform fixed-weight baseline. Existing results therefore do not fully distinguish gains from placing greater weight on accuracy from gains due to adapting weights to each batch’s gradient geometry. A comparison under matched normalization, format gating, and training budgets would help isolate the contribution of dynamic adaptation. (lines 432–435 and lines 458–466) W3. The paper lacks a sensitivity analysis of the key tolerance parameter ε. The auxiliary compatibility constraints use raw gradient inner products, so the interpretation of a fixed ε depends on gradient scale. Although using ε = 0.05 throughout avoids domain-specific tuning, the rationale for this choice and its robustness across auxiliary objectives and domains remain insufficiently characterized. A sensitivity analysis over different ε values would help assess the robustness of the reported performance. (lines 296–315, Table 5) W4. The auxiliary-metric evaluation is incomplete. Auxiliary-metric results are not reported for PCGrad/CAGrad/MGDA or the E/H ablation, limiting assessment of the method’s accuracy–auxiliary trade-off. Reporting these metrics alongside accuracy would help determine whether the observed accuracy gains preserve auxiliary performance. (Tables 1–2 and Figure 5, Figure 6) W5. The online evaluation provides limited method-specific attribution and discussion of serving costs. The A/B test does not report a same-architecture baseline trained with an alternative RL method, limiting attribution of the observed gains specifically to G²RPO rather than the overall deployed system. Table 3 also reports a latency of 138 ms for the proposed system versus 60 ms for the ML baseline. This difference warrants a clearer discussion of measurement comparability and the benefit–cost trade-off, without attributing the inference-latency difference directly to the training-time LP. (Table 3, Appendix G)

Questions for authors

Q1. Theory–implementation consistency: How does the guarantee in Theorem 1 apply after advantage post-processing and AdamW updates? Can you provide evidence on the magnitude of the proxy-gradient error and whether the actual parameter updates satisfy the auxiliary compatibility constraints? Q2. Value of dynamic weighting: How does G²RPO compare with a validation-selected non-uniform fixed-weight baseline under matched normalization, format gating, and training budgets? Would this comparison support an additional benefit from per-batch adaptation rather than simply emphasizing accuracy? Q3. Tolerance sensitivity: How was ε = 0.05 selected, and how sensitive are accuracy and auxiliary metrics to different ε values? How does this tolerance compare with the observed gradient inner-product magnitudes across objectives and domains? Q4. Accuracy–auxiliary trade-off: Can you report auxiliary-metric results for PCGrad/CAGrad/MGDA and the E/H ablation alongside accuracy? Do these comparisons indicate that the accuracy gains are achieved without substantial degradation of auxiliary performance? Q5. Online attribution and serving costs: Is there an available same-architecture RL comparison that helps isolate G²RPO’s contribution to the online gains? Were the 138-ms and 60-ms latencies measured under comparable serving conditions, and how should the CTR improvement be weighed against this latency difference?

3 comments
AuthorsOpenReview ↗

Response to Reviewer 8yMZ (1/2)

We thank the reviewer for the detailed review and the precise questions. [G1]–[G3] refer to the Global Response. Equation numbers follow the revised paper: the LP, Eq. 14 in the submission, is Eq. 13 in the revised paper.

[W1 / Q1] Theory–Implementation Consistency


Takeaway. Theorem 1 is a conditional guarantee for a gradient step along the LP direction and does not automatically extend to the implemented update; the revised paper states this scope, and the realized update has positive first-order accuracy progress in 95.1–97.0% of training steps.

(1) Scope of the Guarantee.

The reviewer is right. Theorem 1 is a conditional guarantee for a gradient step θ + η d⋆ along the LP direction d⋆ = Σₖ w⋆ₖ gₖ, under Assumptions 1–2.

This follows the convention of gradient-based multi-objective methods: the convergence results of PCGrad (Yu et al., 2020; Theorem 1) and CAGrad (Liu et al., 2021; Theorem 3.2) are likewise stated for gradient-descent updates with a bounded step size, while in practice both are paired with adaptive optimizers (e.g., Adam in CAGrad's experiments). Our guarantee does not automatically extend to the implemented update, which applies batch-wide advantage standardization (Algorithm 1) and Paged AdamW (8-bit) (Table 6).

The revised paper states this scope explicitly (Section 3.4 and Appendix G).

(2) Proxy-Gradient Error and Realized Updates.

We do not have per-step measurements of the proxy-gradient error δₖ, or of the auxiliary constraints on the realized updates. For the primary objective, we measured the realized parameter update θₜ₊₁ − θₜ itself, i.e., the update applied by the optimizer after advantage standardization and AdamW (Appendix E, Table 8).

  • Its first-order accuracy progress ⟨θₜ₊₁ − θₜ, g_p⟩ is positive in 95.1–97.0% of training steps.
  • Its mean cosine similarity with the accuracy gradient is 0.66–0.77.
  • On the auxiliary side, the constraints take effect through the implemented optimizer: (I) differs from accuracy-only (E) only in the LP weighting, and it has higher Div@1 in all six domains (e.g., Fashion 0.2968 → 0.3115).

[W2 / Q2] Value of Dynamic Weighting


Takeaway. Conflicts vary across batches, and against a non-uniform fixed-weight baseline G²RPO has higher HR@1 in five domains (the same in Health) and higher Div@1 in five domains.

(1) Why Per-Batch Adaptation.

Both whether a conflict occurs and how strong it is vary across batches (Fig. 1).

  • Accuracy–groundedness: the cosine similarity is negative in 14% of cases and non-negative in the rest, so this conflict appears only in some batches.
  • Accuracy–diversity: the similarity is negative in 97% of cases, but its value is spread over a distribution (mean −0.23) rather than fixed, so the strength of the conflict changes from batch to batch.

A fixed weight vector applies the same mixture to every batch, so it cannot respond to the batches where a conflict appears or is severe. The LP re-solves the weights for each mini-batch.

(2) Non-Uniform Fixed-Weight Baseline.

The baseline fixes the weights to (w_acc, w_div, w_gr) = (0.84, 0.15, 0.01). It differs from G²RPO only in whether the weights are fixed or re-solved for each mini-batch.

Excerpt of Table 3 (HR@1 / Div@1).

MethodFashionClothingGroceryHealthCDsMovieLens
(H) Fixed weights19.77 / 0.29028.98 / 0.277710.45 / 0.27997.14 / 0.113914.44 / 0.23972.76 / 0.0349
(I) G²RPO (per-batch LP)22.21 / 0.311510.46 / 0.296412.12 / 0.32887.14 / 0.118817.58 / 0.23933.29 / 0.0395

Each cell reports HR@1 / Div@1 (variants (H) and (I) of the ablation in Table 3; see [G2] (1)).

  • G²RPO has higher HR@1 than the fixed weights in five domains and the same HR@1 in Health, and higher Div@1 in five domains (CDs: 0.2393 vs 0.2397).
  • The gain is therefore not explained by a fixed emphasis on accuracy alone; this supports an additional benefit from per-batch adaptation.
AuthorsOpenReview ↗

Response to Reviewer 8yMZ (2/2)

[W3 / Q3] Tolerance ε


Takeaway. ε = 0.05 was selected on the Fashion validation split, fixed for all domains, and gives the highest HR@1 of the four values in all six domains.

Please refer to [G1] in the Global Response.

  • ε = 0.05 was selected by grid search over {0, 0.01, 0.05, 0.1} on the Fashion validation split and fixed for all domains.
  • It gives the highest HR@1 of the four in all six domains ([G1] (2)).
  • Div@1 at ε = 0.05 stays within 0.012 of the best value among the four settings in every domain.
  • The gradient-scale analysis in [G1] (3) shows that the median auxiliary-to-accuracy gradient-norm ratios are 0.62–0.81 and that cosine-normalizing the Gram matrix changes HR@1 by at most 0.12.

[W4 / Q4] Accuracy–Auxiliary Trade-off


Takeaway. The full method is higher than accuracy-only (E) on both HR@1 and Div@1 in all six domains, and its Div@1 is at most 0.014 below the best of PCGrad, CAGrad, and MGDA.

Please refer to [G2] in the Global Response.

  • E/H ablation. [G2] (1) reports HR@1 / Div@1 for all ablation variants on all six domains. The full method ((H) in the submission, (I) in the revision) is higher than accuracy-only (E) on both HR@1 and Div@1 in all six domains (e.g., Fashion 22.21 / 0.3115 vs 15.01 / 0.2968).
  • PCGrad / CAGrad / MGDA. [G2] (3) reports their Div@1 alongside HR@1. G²RPO has the highest HR@1 in four domains and the highest or tied-highest Div@1 in three. Across all six domains, its Div@1 is at most 0.014 below the best of the three baselines.

The LLM-judge score is not reported for rows (E) and (H) or for PCGrad/CAGrad/MGDA: the commercial LLM used as the judge is no longer served through the API.


[W5 / Q5] Online Attribution and Serving Costs


Takeaway. The latencies were measured on the same hardware, the gap comes from model size and rationale generation rather than the LP, and the online claim is scoped to deployability and effectiveness against production baselines.

Please refer to [G3] (3)–(4) in the Global Response.

  • The 60 ms and 138 ms latencies were measured on the same hardware under the same serving conditions.
  • As the reviewer notes, the LP is a training-time component and adds no computation at inference; the gap comes from the larger model size and from generating the recommendation rationale.
  • 138 ms is within our 150 ms latency budget, for a +0.16pp (+2.2% relative) CTR gain over the strongest production baseline plus a user-facing rationale.

No same-architecture RL baseline is available online, so we scope the online claim to deployability and effectiveness against production baselines; method-specific attribution rests on the offline comparisons (Table 1, [G2]).

Dear Reviewer 8yMZ,

The authors have provided detailed reponse. Could you please read the response and give the feedback whether the rebuttal has addressed your concern?

Thanks.

Comments on the paper

5 comments
AuthorsOpenReview ↗

Global Response (3/3)

[G3] Online A/B Test (8yMZ W5/Q5, 8EFq Q3/Q4/Q5)


Takeaway. The +0.16pp CTR gain over ML remains significant with users as the clustering unit, the latency gap is not caused by the LP, and the online claim is scoped to deployability and effectiveness against production baselines.

(1) User-Clustered Uncertainty.

Assignment was by deterministic user-level hashing, so repeated impressions from the same user can be correlated. We therefore re-estimate the uncertainty with users as the clustering unit (10,000 bootstrap resamples), keeping the impression-weighted CTR definition.

User-clustered bootstrap (10K resamples).

ComparisonΔCTR95% CI (user-clustered bootstrap, 10K)p
vs ML+0.16pp[+0.045, +0.271]0.008
vs SASRec+3.70pp[+3.58, +3.82]<0.001

The gain over the strongest production baseline (ML) remains significant with users as the clustering unit.


(2) Deployed Variant.

  • Kept: the deployed model is trained with the same G²RPO procedure as the offline models.
  • Changed (data): it is trained on the interaction data of the production service instead of the public datasets.
  • Changed (candidates): the candidate list in the prompt is supplied by the production retrieval stage (top-N) instead of the offline 200-item sampled set.
  • Changed (post-filter): outputs pass a business-rule post-filter that is applied identically to all models in the test.

(3) Latency.

  • The 60 ms (ML) and 138 ms (ours) latencies were measured on the same hardware under the same serving conditions.
  • The LP is solved only during training to set the reward weights; it requires no additional computation at inference, so the latency gap is not caused by the LP.
  • The gap comes from the larger model size and from generating the recommendation rationale. The changes specific to the deployed variant (retrieval-supplied candidates and the business-rule post-filter) have little effect on latency.
  • 138 ms is within our internal 150 ms latency budget. The trade-off is +0.16pp absolute (+2.2% relative) CTR over the strongest production baseline, plus a user-facing rationale for each recommendation.

(4) Same-Architecture RL Baseline Online.

No same-architecture RL baseline (e.g., GRPO or GDPO on the same backbone) is available from the online test.

We therefore scope the online claim to deployability and effectiveness against production baselines, not to isolating G²RPO from GRPO/GDPO online. Method-specific attribution rests on the offline comparisons (Table 1 and the ablation in Table 3).

In the revised paper, Section 4.4 reports the user-clustered CIs, and Appendix I ("Online A/B Test Details") gives the deployed-variant description and the same-hardware statement and scopes the online claim as above.

AuthorsOpenReview ↗

Global Response (2/3)

[G2] Six-Domain Ablation with Auxiliary Metrics, Matched SFT, and Auxiliary Metrics of PCGrad/CAGrad/MGDA (8yMZ W4/Q4, 8EFq Q6)

Takeaway. On all six domains, the full method (I) is best in 15 of the 18 domain–metric columns and second-best in the remaining three.

(1) Ablation on All Six Domains with Accuracy and Auxiliary Metrics.

Table 3 replaces the two-domain ablation figure (Fig. 6 in the submission) with all six domains and tabulated numbers, and reports each variant's auxiliary metrics alongside accuracy. Variant letters follow the revised manuscript: the fixed-weight baseline is inserted as (H), and the full method, which was (H) in the submission, becomes (I).

Table 3 (revised paper): Ablation on all six domains.

VariantFashionClothingGroceryHealthCDsMovieLens
(A) Base (Gemma-3-4B-it)2.15 / 0.1774 / 1.581.88 / 0.2455 / 1.611.42 / 0.2101 / 1.550.95 / 0.0894 / 1.491.73 / 0.1802 / 1.640.42 / 0.0241 / 1.42
(B) +SFT12.94 / 0.2032 / 2.288.32 / 0.2948 / 2.3010.28 / 0.2447 / 2.254.63 / 0.1012 / 2.1910.85 / 0.2130 / 2.333.02 / 0.0320 / 2.11
(C) +SFT+GRPO9.75 / 0.2287 / 2.359.71 / 0.2871 / 2.3611.05 / 0.2688 / 2.332.91 / 0.1078 / 2.2713.20 / 0.2158 / 2.401.85 / 0.0348 / 2.18
(D) +SFT+GDPO12.61 / 0.2410 / 2.418.15 / 0.3005 / 2.437.30 / 0.2905 / 2.394.65 / 0.1132 / 2.349.42 / 0.2205 / 2.461.32 / 0.0372 / 2.24
(E) +SFT+G²RPO (accuracy-only, w = e_p + hard gate)15.01 / 0.2968 / –10.11 / 0.2956 / –11.32 / 0.2872 / –5.11 / 0.0978 / –15.98 / 0.2229 / –2.40 / 0.0385 / –
(F) +SFT+G²RPO (LP + format-as-reward)20.15 / 0.2905 / 2.6810.02 / 0.2930 / 2.6511.62 / 0.3154 / 2.626.72 / 0.1160 / 2.5816.04 / 0.2311 / 2.702.94 / 0.0381 / 2.49
(G) +SFT+G²RPO (trainable w + hard gate)16.88 / 0.3087 / 2.559.85 / 0.2902 / 2.5711.30 / 0.3041 / 2.526.41 / 0.1224 / 2.4714.92 / 0.2240 / 2.592.11 / 0.0302 / 2.38
(H) +SFT+G²RPO (fixed weights + hard gate)19.77 / 0.2902 / –8.98 / 0.2777 / –10.45 / 0.2799 / –7.14 / 0.1139 / –14.44 / 0.2397 / –2.76 / 0.0349 / –
(I) +SFT+G²RPO (full)22.21 / 0.3115 / 2.8710.46 / 0.2964 / 2.8512.12 / 0.3288 / 2.817.14 / 0.1188 / 2.7917.58 / 0.2393 / 2.883.29 / 0.0395 / 2.72

Each cell reports HR@1 / Div@1 / mean LLM-judge rationale score (0–3), averaged over 3 seeds. For each metric, the best value in a domain is in bold and the second-best in italics.

  • The full method (I) is best in 15 of the 18 domain–metric columns and second-best in the remaining three.
  • (E) vs (I), which differ only in whether w is fixed to e_p or chosen by the LP: (I) is higher on both HR@1 and Div@1 in all six domains.
  • (H) vs (I), which differ only in whether the weights are fixed or re-solved for each mini-batch: (I) has higher HR@1 in five domains and the same HR@1 in Health, and higher Div@1 in five domains (CDs: 0.2393 vs 0.2397).

The LLM-judge score is not reported for rows (E) and (H): the commercial LLM used as the judge is no longer served through the API (gpt-4o).


(2) Matched SFT Starting Point.

Row (B) of Table 3, which is in the main text, is the matched SFT starting point: the same SFT checkpoint is the common initialization of all RL-based variants and baselines.


(3) Auxiliary Metrics of PCGrad, CAGrad, and MGDA.

Table 1 reports only accuracy for these three baselines; Table 7 in Appendix C adds Div@1 alongside HR@1.

Table 7 (revised paper): HR@1 / Div@1 against PCGrad, CAGrad, and MGDA.

DomainPCGradCAGradMGDAG²RPO
Fashion20.90 / 0.304720.02 / 0.276518.35 / 0.278922.21 / 0.3115
Clothing10.90 / 0.31019.61 / 0.288810.20 / 0.289610.46 / 0.2964
Grocery11.14 / 0.267510.36 / 0.289110.54 / 0.254112.12 / 0.3288
Health6.22 / 0.11886.58 / 0.09816.71 / 0.10047.14 / 0.1188
CDs15.70 / 0.247015.94 / 0.211315.16 / 0.204617.58 / 0.2393
MovieLens2.93 / 0.03753.11 / 0.02643.75 / 0.03973.29 / 0.0395

Each cell reports HR@1 / Div@1; for each metric, the best value in a domain is in bold and the second-best in italics.

  • G²RPO has the highest HR@1 in four domains and the highest or tied-highest Div@1 in three.
  • Across all six domains, its Div@1 is at most 0.014 below the best of the three baselines.

The LLM-judge score is not reported for these baselines, for the same reason as in (1).

AuthorsOpenReview ↗

Global Response (1/3)

Global Response

We thank the AC and all reviewers for their careful reading. Issues shared across reviewers are addressed in [G1]–[G3] below; the per-reviewer responses are posted as separate comments and refer to these items. All table, figure, section, and appendix numbers refer to the revised paper uploaded with this response, in which the main additions are marked in red.


[G1] Tolerance ε: Value Used, Sensitivity, Gradient Scale, and Fallback (8yMZ W3/Q3, 8EFq Q1/Q2)

Takeaway. All reported G²RPO results outside the ε sweep use ε = 0.05, selected once on the Fashion validation split; it gives the highest HR@1 in all six domains, and the fallback is used in 0.6% of steps.

(1) Value Used and Selection.

  • All reported G²RPO results use ε = 0.05 (Table 6); the only exception is the ε sweep in (2).
  • The 0.0 in the released example config is a default value that was not used for any reported run.
  • ε was selected once, by grid search over {0, 0.01, 0.05, 0.1} on the Fashion validation split (the second-most-recent interaction under leave-one-out). It was then fixed for all six domains.

(2) Sensitivity to ε.

Table 10 (revised paper): Sensitivity to ε.

ε00.010.05 (used)0.1
Fashion HR@122.0522.1422.2121.87
Fashion Div@10.32190.31610.31150.3001
Fashion Div@50.32810.28880.28500.2644
Clothing HR@19.569.7710.469.21
Clothing Div@10.30800.29110.29640.3010
Clothing Div@50.30500.29940.30200.3198
Grocery HR@110.1010.7812.1210.33
Grocery Div@10.33870.32770.32880.2933
Grocery Div@50.34770.33930.33480.3121
Health HR@15.776.127.146.33
Health Div@10.11450.11510.11880.1067
Health Div@50.09670.10130.12080.1185
CDs HR@114.7615.3217.5815.88
CDs Div@10.24330.24560.23930.2213
CDs Div@50.25470.25710.23940.2431
MovieLens HR@12.112.683.292.88
MovieLens Div@10.03970.03330.03950.0412
MovieLens Div@50.04010.03560.03720.0400
Fallback used4.7%2.1%0.6%0.2%

For each metric, the best value is in bold and the second-best is in italics. The fallback rate is aggregated over all domains.

  • ε = 0.05 gives the highest HR@1 in all six domains.
  • On the auxiliary side, Div@1 at ε = 0.05 stays within 0.012 of the best value among the four settings in every domain.
  • A larger ε reduces how often the fallback is used (4.7% → 0.2%).

(3) Gradient Scale.

The reviewers are right: the LP uses raw inner products, so the meaning of a fixed ε depends on the gradient scale. Per-reward group normalization (Eq. 4) aligns the relative scale across rewards, but it does not remove this absolute-scale dependence.

Table 11 (revised paper): Gradient-norm ratios and the effect of cosine-normalizing the Gram matrix (Fashion and Health).

FashionHealth
Median ‖g_div‖ / ‖g_acc‖0.620.67
Median ‖g_gr‖ / ‖g_acc‖0.810.78
HR@1, raw inner product (used)22.217.14
HR@1, cosine-normalized Gram22.097.02

No auxiliary reward dominates the LP by norm alone (median ratios 0.62–0.81), and cosine-normalizing the Gram matrix changes HR@1 by at most 0.12.


(4) Fallback Frequency and Zero Accuracy Gradients.

When the solver returns no solution for Eq. 13, the auxiliary constraints are dropped one at a time (groundedness first, then diversity) and the LP is re-solved; if this also fails, the primary-only update w⋆ = e_p is used. At ε = 0.05, this fallback (constraint relaxation and primary-only combined) is used in 0.6% of steps (last row of the table in (2)).

Separately, we report how often the accuracy gradient is zero under the binary accuracy reward (G = 16).

Table 12 (revised paper): Zero accuracy gradients.

FashionClothingGroceryHealthCDsMovieLens
Groups with zero accuracy gradient18.9%29.6%25.1%27.4%20.7%38.4%
Mini-batches with zero accuracy gradient (B = 4)0.4%1.8%1.0%1.2%0.5%4.3%

g_p is aggregated over the B = 4 groups of a mini-batch, so it vanishes only when all four groups are degenerate at once.

In the revised paper, the fallback rule is stated in Section 3.4, and these results are in Appendix F ("Robustness of the Alignment-Aware Weighting"): the ε-sensitivity table is Table 10, the gradient-scale table is Table 11, and the zero-accuracy-gradient table is Table 12.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.