- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper tackles multi-reward RLVR for generative recommendation. Accuracy is treated as the primary reward; diversity and rationale groundedness serve as auxiliaries. The core idea is to normalize each reward separately, then solve a simplex-constrained LP that keeps the update aligned with the accuracy gradient while capping first-order interference from the auxiliary gradients. A format-compliance gate filters training samples. Results span five Amazon domains, MovieLens, a rationale-quality evaluation, and an online A/B test.
Critical strengths
- The gradient conflict analysis is convincing: accuracy and diversity gradients frequently oppose each other, and Figure 2 shows concrete cases where GRPO and GDPO actually hurt accuracy compared to SFT. This motivates the accuracy-first design well.
- The experimental coverage is broad — conventional and generative recommenders, alternative gradient-combination baselines, ablations, rationale quality, plus a production deployment. The weighting module's overhead looks small in the tested setting.
Critical weaknesses
-
I cannot tell how much the LP itself adds over simpler alternatives. Figure 6 compares accuracy-only training with the full method, but only on two domains and without tabulated numbers; the main table also misses a matched SFT starting point. The optimization setup also needs clarification: the tolerance operates on scale-dependent gradient inner products, yet the paper says ε = 0.05 while the example config uses 0.0, and fallback statistics from the implementation are not reported. Without these details, it is hard to assess both the LP's marginal benefit and its actual behavior during training.
-
Several choices in the online experiment are left unexplained. The deployed variant's modifications relative to the offline method are not detailed, the CTR difference lacks a user-clustered confidence interval, and the latency gap (60 ms vs. 138 ms) is reported without confirming matched hardware. Taken together, these gaps weaken the production evidence.
Formatting. The spacing around headings and figures looks artificially compressed, especially pp. 8–9; the template margins may have been modified. Small tables and tightly wrapped text on those pages are hard to read.
Questions for authors
- Given that the paper states ε = 0.05 but the example config uses 0.0, which value was actually used in the reported experiments? How sensitive are the weights and the accuracy/auxiliary trade-off to this tolerance?
- What do the fallback logs show? How often does constraint relaxation or primary-only fallback kick in, and how frequently is the primary gradient effectively zero?
- What was changed for the deployed variant, and which parts of the offline method were kept?
- Can you estimate uncertainty for the CTR difference using users as the clustering unit while keeping the impression-weighted CTR definition?
- Were the 60 ms and 138 ms latencies measured under the same hardware and serving conditions? What explains the gap, including any changes specific to the deployed variant?
- Could you add the matched SFT baseline to the main table, and extend the Figure 6 comparison to more domains with tabulated numbers?
4 comments
Response to Reviewer 8EFq (2/2)
[W: Marginal Benefit of the LP]
Takeaway. On the identical pipeline, G²RPO is above PCGrad, CAGrad, and MGDA on every Table 1 entry except three, and the full method is higher than accuracy-only (E) on both HR@1 and Div@1 in all six domains.
(1) Simpler Alternatives on the Identical Pipeline.
PCGrad, CAGrad, and MGDA are run on exactly the same training pipeline as G²RPO: the same K = 3 per-reward gradient proxies from group-normalized, format-gated advantages, with only the LP weighting replaced by their update rules (Appendix C).
In Table 1 (HR@1 / HR@5 / NDCG@5 on six domains), G²RPO is above all three on every entry except:
- Clothing HR@1 (PCGrad 10.90 vs 10.46);
- CDs HR@5 (MGDA 37.63 vs 36.54);
- MovieLens HR@1 (MGDA 3.75 vs 3.29).
(2) Accuracy-Only and Fixed Weights vs Full.
The most direct tests of the LP compare the full method (I) with accuracy-only (E), where w is fixed to e_p, and with fixed weights (H), where (w_acc, w_div, w_gr) = (0.84, 0.15, 0.01) for every mini-batch (variant letters as in Table 3 of the revised paper).
Excerpt of Table 3 (HR@1 / Div@1).
| Variant | Fashion | Clothing | Grocery | Health | CDs | MovieLens |
|---|---|---|---|---|---|---|
| (E) accuracy-only | 15.01 / 0.2968 | 10.11 / 0.2956 | 11.32 / 0.2872 | 5.11 / 0.0978 | 15.98 / 0.2229 | 2.40 / 0.0385 |
| (H) fixed weights | 19.77 / 0.2902 | 8.98 / 0.2777 | 10.45 / 0.2799 | 7.14 / 0.1139 | 14.44 / 0.2397 | 2.76 / 0.0349 |
| (I) full (per-batch LP) | 22.21 / 0.3115 | 10.46 / 0.2964 | 12.12 / 0.3288 | 7.14 / 0.1188 | 17.58 / 0.2393 | 3.29 / 0.0395 |
Each cell reports HR@1 / Div@1.
- vs (E): the full method is higher on both HR@1 and Div@1 in all six domains.
- vs (H): it has higher HR@1 in five domains and the same HR@1 in Health, and higher Div@1 in five domains (CDs: 0.2393 vs 0.2397).
The tight spacing still makes parts of the revised paper hard to read. I’ll leave template compliance to the AC and base my rating on the technical results.
I appreciate the additional work. The fixed-weight comparison and the user-clustered CTR analysis address my main concerns, so I’m raising my rating from 2 to 3.
Thank you so much for your follow-up and for raising your score! We will make sure to refine the spacing and readability in the final version.
Response to Reviewer 8EFq (1/2)
We thank the reviewer for the careful review and the concrete questions. [G1]–[G3] refer to the Global Response.
[Q1] Value of ε, Config, and Sensitivity
Takeaway. All reported G²RPO results outside the ε sweep use ε = 0.05, the 0.0 in the example config was not used for any reported run, and 0.05 gives the highest HR@1 of the four values in all six domains.
Please refer to [G1] (1)–(3) in the Global Response.
The reviewer is right that the tolerance acts on scale-dependent raw inner products. The median auxiliary-to-accuracy gradient-norm ratios are 0.62–0.81, and cosine-normalizing the Gram matrix changes HR@1 by at most 0.12 ([G1] (3)).
[Q2] Fallback Logs and Zero Primary Gradient
Takeaway. Constraint relaxation and the primary-only fallback combined occur in 0.6% of steps at ε = 0.05, and the accuracy gradient g_p is zero in 0.4–4.3% of mini-batches.
Please refer to [G1] (4) in the Global Response.
[Q3] Deployed Variant
Takeaway. The deployed model keeps the same G²RPO training procedure but uses production interaction data, retrieval-supplied candidates, and a business-rule post-filter applied identically to all models.
Please refer to [G3] (2) in the Global Response.
[Q4] User-Clustered Uncertainty
Takeaway. Yes: with users as the clustering unit, the +0.16pp CTR gain over ML has a 95% CI of [+0.045, +0.271] (p = 0.008).
Please refer to [G3] (1) in the Global Response.
With users as the clustering unit (10,000 bootstrap resamples) and the impression-weighted CTR definition unchanged:
User-clustered bootstrap (10K resamples).
[Q5] Latency
Takeaway. Yes: the 60 ms and 138 ms latencies were measured on the same hardware, and the latency gap is not caused by the LP.
Please refer to [G3] (3) in the Global Response.
[Q6] Matched SFT Baseline and Extended Fig. 6
Takeaway. Row (B) +SFT of Table 3 is the matched SFT starting point, and Table 3 replaces the Fig. 6 comparison with all six domains.
Please refer to [G2] (1)–(2) in the Global Response.