- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Group Reward-Mass Optimization (GRMO) reframes KL-regularized reward maximization as matching a policy to a reward-tilted target distribution. It samples groups of candidate recommendation lists, constructs reward-aware probability masses with behavior-policy correction, and minimizes a groupwise reverse KL. In its practical objective, rewards are assigned to items at their original list positions and probability matching is performed separately at each position. Experiments on Reddit-v2 with Qwen2.5-0.5B and Llama-3.2-3B report ranking improvements and better query-conditioned access to high-quality recommendations.
Critical strengths
-
The objective addresses a relevant limitation of reward-maximizing updates. It explicitly models how probability should be distributed among a group of candidate responses, rather than updating each sampled response only through its relative advantage.
-
The reward design is tailored to ranked recommendation lists. Item credit is kept at its original rank, and invalid, previously seen, or unmatched recommendations do not transfer their reward to other positions.
-
The target distribution is controlled. Reference-policy mass anchors the target, while a target-KL budget limits how far reward tilting can move it.
-
The evaluation considers more than ranking metrics. The authors measure within-query consistency, between-query separation, and access to high-quality sampled trajectories, and test different decoding temperatures.
Critical weaknesses
- The novelty relative to prior distribution-matching methods needs sharper definition. Reward-tilted target distributions and KL projection have been studied in other settings, including LPO https://arxiv.org/abs/2605.06139; reward-aligned probability distributions for list recommendation also have precedent in GFN4Rec https://arxiv.org/abs/2306.02239. GRMO’s recommendation-specific contribution appears to be its sampled-group, position-level reward and matching design, but the paper does not directly compare against these closest approaches.
- The practical objective is a local groupwise surrogate, not matching the full response distribution. It matches probabilities only among the candidates sampled for a group and separately at each recommendation position. High-quality responses outside the sampled support receive no direct target mass, and the position-wise loss does not fully model dependencies across the complete list.
- The ranking gains over the strongest baseline are modest, and evaluation is limited to one benchmark. On Reddit-v2, the reported gains over Rank-GRPO are below 1.2% on R@20 and NDCG@20 for both model sizes. The results do not establish generalization to other recommendation domains.
- The diversity evidence comes from stochastic sampling, while ranking is evaluated with greedy decoding. The diversity analysis uses repeated sampled responses for a fixed set of queries. It remains unclear how much of this benefit would translate to systems that primarily serve deterministic recommendations.
Questions for authors
Please refer to Weaknesses.
7 comments
2. Response to Weakness 2: Local Groupwise & Position-Wise Surrogate, Cross-Position Dependencies, and Rank-Band Gains
Thank you for this incisive theoretical observation. While exact population KL minimization over full lists is intractable, GRMO's rank-local surrogate preserves list-wide coherence and improves both head and tail positions for two reasons:
- Autoregressive Conditioning Preserves Cross-Position Dependencies: Although group softmax normalization is computed at each rank , every item score
is strictly conditioned on the dialog context and the entire prefix of previously generated items (ranks ). Thus, the policy preserves inter-item dependencies such as non-duplication and complementary diversity: under ordinary greedy decoding (
repetition_penalty=1.0, zero post-filtering), converged GRMO achieves>98.5%within-list unique titles. - Broad Gain Attribution Across All List Positions: Decomposing Llama-3.2-3B's
NDCG@20gain by rank band (Table 2) demonstrates that GRMO improves not only Rank 1 (+0.00088, 48.5% of total gain), but also middle Ranks 6–10 (+0.00045, 24.7%) and tail Ranks 11–20 (+0.00056, 30.9%):
| Recommendation Rank Band | 0.1283 Baseline Contribution | Converged GRMO Contribution | Net Band Gain | Share of Total Gain |
|---|---|---|---|---|
| Rank 1 (Top-1 item) | 0.03921 | 0.04009 | +0.00088 | 48.5% |
| Ranks 2–5 | 0.04840 | 0.04833 | -0.00007 | -4.0% |
| Ranks 6–10 (Mid-list) | 0.02191 | 0.02236 | +0.00045 | 24.7% |
| Ranks 11–20 (Tail-list) | 0.01878 | 0.01934 | +0.00056 | 30.9% |
| Total NDCG@20 | 0.12830 | 0.13012 (0.1301) | +0.00182 | 100% |
3. Response to Weakness 3: Substantially Larger Gains Upon Full Convergence (+1.42% to +3.71%) and Stratified Generalization
3.1 Full Convergence Surpasses the 1.2% Margin Across Both Backbones (Winning 16/16 Metrics)
The checkpoints reported in our initial submission had not yet fully converged due to limited training epochs prior to the submission deadline (resulting in <1.2% gains). After continuing training to full convergence under the exact same 10,972-row benchmark and greedy decoding, the converged results are reported below: GRMO significantly outperforms Rank-GRPO (exp∞) across all 16/16 metrics on both backbones, decisively surpassing the 1.2% margin (Table 3):
| Backbone / Method | R@5 | R@10 | R@15 | R@20 | NDCG@5 | NDCG@10 | NDCG@15 | NDCG@20 |
|---|---|---|---|---|---|---|---|---|
| Qwen2.5-0.5B | ||||||||
| Rank-GRPO () | 0.0946 | 0.1468 | 0.1813 | 0.2047 | 0.0744 | 0.0915 | 0.1016 | 0.1079 |
| GRMO (Fully Converged) | 0.0981 | 0.1478 | 0.1830 | 0.2087 | 0.0762 | 0.0924 | 0.1027 | 0.1096 |
| Relative Gain (vs. Paper Table 1) | +3.71% | +0.67% | +0.96% | +1.97% | +2.45% | +0.96% | +1.11% | +1.60% |
| Paired Bootstrap (vs. Official Dump) | p=0.0029 | +0.93% | +0.97% | +1.07% | p=0.0103 | +1.10% | p=0.052 | p=0.0340 |
| Llama-3.2-3B | ||||||||
| Rank-GRPO () | 0.1178 | 0.1756 | 0.2150 | 0.2368 | 0.0919 | 0.1107 | 0.1223 | 0.1283 |
| GRMO (Fully Converged) | 0.1187 | 0.1774 | 0.2176 | 0.2419 | 0.0928 | 0.1119 | 0.1236 | 0.1301 |
| Relative Gain (vs. Paper Table 1) | +0.80% | +1.03% | +1.19% | +2.14% | +0.95% | +1.09% | +1.06% | +1.42% |
| Paired Bootstrap (vs. 0.1283 Dump) | - | - | p=0.052 | p=0.0017 | - | p=0.0037 | p=0.0010 | p=0.0001 |
- Llama-3.2-3B:
0.1301NDCG@20 (+1.42%, Row 95% CI[+0.00094, +0.00271], ) and0.2419Recall@20 (+2.14%, Row 95% CI[+0.00133, +0.00620], ); 3-seed mean:0.1296 ± 0.0004. - Qwen2.5-0.5B:
0.1096NDCG@20 (+1.60%vs.0.1079;+1.18%with Row 95% CI[+0.00009, +0.00247], even vs. the re-evaluated official checkpoint0.10834),0.2087Recall@20 (+1.97%), and+3.71%Recall@5 (); 3-seed mean:0.1092 ± 0.0004.
3.2 Benchmark Scope & Stratified Generalization Across Recommendation Regimes
Following Rank-GRPO, our work targets multi-turn natural-language conversational list recommendation, where Reddit-v2 is currently the only public large-scale benchmark combining multi-turn natural-language requests (382,829 train pairs; 10,972 test rows) with genuine one-to-many ground-truth annotations (unlike item-ID click logs in MovieLens/Amazon). Beyond validating across two distinct model families spanning a 6$\times$ parameter scale, we evaluated generalization across five recommendation strata varying in intent multiplicity () and seen-history sparsity (Table 4, Llama-3.2-3B):
| Recommendation Stratum | Rows | Baseline NDCG@20 | Converged GRMO NDCG@20 | NDCG@20 | Baseline Recall@20 | Converged GRMO Recall@20 | Recall@20 |
|---|---|---|---|---|---|---|---|
| Single-target intent ($ | GT | =1$) | 7,182 | 0.11702 | 0.11899 | +0.00197 | 0.24798 |
| Moderate multi-intent ($2 \le | GT | \le 4$) | 2,623 | 0.14241 | 0.14408 | +0.00167 | 0.23504 |
| Broad multi-intent ($ | GT | \ge 5$) | 1,167 | 0.16597 | 0.16716 | +0.00119 | 0.18446 |
Cold-history dialogs (seen = 0) | 4,002 | 0.11938 | 0.12209 | +0.00271 (+2.27%) | 0.22170 | 0.22672 | +0.00502 (+2.26%) |
Warm-history dialogs (seen >= 1) | 6,970 | 0.13342 | 0.13472 | +0.00131 (+0.98%) | 0.24757 | 0.25057 | +0.00301 (+1.21%) |
GRMO achieves positive NDCG@20 and Recall@20 gains across 100% (5/5) of strata, including +2.27% NDCG@20 on cold-start conversations (seen = 0). We will also explicitly note cross-domain conversational extensions in our Limitations section.
4. Response to Weakness 4: Translation of Diversity Benefits to Deterministic ( Greedy) Serving
This is an excellent practical question. To directly measure how GRMO benefits production systems that serve purely deterministic ( greedy) recommendations, we evaluated population-level catalog coverage, effective catalog size , Gini popularity concentration, and unique ground-truth item recovery directly on the deterministic greedy predictions across all 2,050 unique test queries on both backbones (Table 5):
| Deterministic ( Greedy) Serving Metric | Qwen: Official Rank-GRPO (exp∞) | Qwen: Converged GRMO | Llama: 0.1283 Baseline | Llama: Converged GRMO |
|---|---|---|---|---|
| Catalog-valid unique items @5 () | 1,660 | 1,776 (+116) | 1,989 | 2,027 (+38) |
| Catalog-valid unique items @10 () | 2,362 | 2,560 (+198) | 2,884 | 2,931 (+47) |
| Catalog-valid unique items @20 () | 3,301 | 3,473 (+172) | 3,860 | 3,985 (+125) |
| Effective catalog size @20 () | 1,283.8 | 1,436.2 (+11.87%) | 1,578.7 | 1,644.6 (+4.18%) |
| Gini concentration index @20 () | 0.7067 | 0.6927 (-0.0140) | 0.6830 | 0.6809 (-0.0021) |
| Between-query Jaccard distance @20 () | 0.9849 | 0.9870 | 0.9885 | 0.9887 |
| Distinct ground-truth items recovered @20 () | 1,267 | 1,331 (+64 movies) | 1,567 | 1,593 (+26 movies) |
Why training-time mass matching translates directly to greedy serving:
Under sparse binary feedback, standard advantage maximization pushes the argmax across many distinct user contexts toward a narrow set of globally popular blockbusters. Because GRMO matches reference-anchored intra-group probability mass during training, head blockbusters are prevented from cannibalizing context-specific niche items. Consequently, even under purely deterministic greedy decoding, distinct user queries peak on distinct context-appropriate items, expanding effective catalog utilization by +11.87% (Qwen) and +4.18% (Llama), lowering Gini concentration, and recovering +64 (Qwen) and +26 (Llama) additional unique ground-truth movies. We will add Table 5 to Section 4.2.
Thank you again for your constructive review; we hope these theoretical clarifications and new empirical findings address all of your concerns.
Official Response to Reviewer HMDu
Submission 58300: Group Reward-Mass Optimization for Generative Recommendation (GRMO)
Dear Reviewer HMDu,
We sincerely thank you for your insightful and technically precise review. We appreciate your recognition of GRMO's core strengths, including explicitly modeling probability allocation across candidate groups rather than relying on uncoupled advantage updates, keeping recommendation credit at its original rank without transferring reward across invalid or seen items, anchoring target tilting to the reference policy, and evaluating query-conditioned path accessibility across temperatures.
To address the four insightful concerns you raised: (1) novelty definition and comparison relative to LPO and GFN4Rec, (2) cross-position dependency modeling under the finite-group position-wise surrogate, (3) ranking gain magnitude over the strongest baseline and benchmark generality, and (4) how stochastic diversity advantages translate into practical benefits for deterministic (T=0 greedy) serving, we conducted targeted theoretical and empirical analyses during the rebuttal period and report them in detail below.
1. Response to Weakness 1: Sharper Novelty Definition Relative to LPO and GFN4Rec
We agree that explicitly contrasting GRMO against LPO and GFN4Rec sharpens our contribution, and we will add Table 1 and a dedicated discussion to Sections 2.2-2.3:
+0.00072to+0.00090NDCG@20, while our unowned-token reference anchor boosts Exact-20 format validity from51.85%to98.94%.