acceptodds
Under review as a conference paper at ICLR 2027

Group Reward-Mass Optimization for Generative Recommendation

Abstract

Generative recommendation formulates personalized recommendation as a conditional sequence generation task, allowing models to directly generate items aligned with user preferences. Because a user's intent can correspond to multiple valid items, sparse-reward policy optimization can concentrate generation probability on a narrow set of outputs, reducing recommendation diversity and coverage. To address this issue, we propose Group Reward-Mass Optimization (GRMO), a framework that formulates policy learning as direct probability mass matching within candidate groups. GRMO leverages recommendation rewards to construct a target probability mass distribution over sampled candidates. By performing intra-group normalization on both the target and current policy distributions, GRMO minimizes their KL divergence, directly optimizing the relative probability allocation. Experiments on Reddit-v2 show that GRMO improves ranking performance over GRPO while producing more stable, query-conditioned recommendations and preserving access to high-quality alternatives.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
RL, Generative Recommendation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_HMDuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Group Reward-Mass Optimization (GRMO) reframes KL-regularized reward maximization as matching a policy to a reward-tilted target distribution. It samples groups of candidate recommendation lists, constructs reward-aware probability masses with behavior-policy correction, and minimizes a groupwise reverse KL. In its practical objective, rewards are assigned to items at their original list positions and probability matching is performed separately at each position. Experiments on Reddit-v2 with Qwen2.5-0.5B and Llama-3.2-3B report ranking improvements and better query-conditioned access to high-quality recommendations.

Critical strengths
  • The objective addresses a relevant limitation of reward-maximizing updates. It explicitly models how probability should be distributed among a group of candidate responses, rather than updating each sampled response only through its relative advantage.

  • The reward design is tailored to ranked recommendation lists. Item credit is kept at its original rank, and invalid, previously seen, or unmatched recommendations do not transfer their reward to other positions.

  • The target distribution is controlled. Reference-policy mass anchors the target, while a target-KL budget limits how far reward tilting can move it.

  • The evaluation considers more than ranking metrics. The authors measure within-query consistency, between-query separation, and access to high-quality sampled trajectories, and test different decoding temperatures.

Critical weaknesses
  • The novelty relative to prior distribution-matching methods needs sharper definition. Reward-tilted target distributions and KL projection have been studied in other settings, including LPO https://arxiv.org/abs/2605.06139; reward-aligned probability distributions for list recommendation also have precedent in GFN4Rec https://arxiv.org/abs/2306.02239. GRMO’s recommendation-specific contribution appears to be its sampled-group, position-level reward and matching design, but the paper does not directly compare against these closest approaches.
  • The practical objective is a local groupwise surrogate, not matching the full response distribution. It matches probabilities only among the candidates sampled for a group and separately at each recommendation position. High-quality responses outside the sampled support receive no direct target mass, and the position-wise loss does not fully model dependencies across the complete list.
  • The ranking gains over the strongest baseline are modest, and evaluation is limited to one benchmark. On Reddit-v2, the reported gains over Rank-GRPO are below 1.2% on R@20 and NDCG@20 for both model sizes. The results do not establish generalization to other recommendation domains.
  • The diversity evidence comes from stochastic sampling, while ranking is evaluated with greedy decoding. The diversity analysis uses repeated sampled responses for a fixed set of queries. It remains unclear how much of this benefit would translate to systems that primarily serve deterministic recommendations.
Questions for authors

Please refer to Weaknesses.

7 comments

Official Response to Reviewer HMDu

Submission 58300: Group Reward-Mass Optimization for Generative Recommendation (GRMO)


Dear Reviewer HMDu,

We sincerely thank you for your insightful and technically precise review. We appreciate your recognition of GRMO's core strengths, including explicitly modeling probability allocation across candidate groups rather than relying on uncoupled advantage updates, keeping recommendation credit at its original rank without transferring reward across invalid or seen items, anchoring target tilting to the reference policy, and evaluating query-conditioned path accessibility across temperatures.

To address the four insightful concerns you raised: (1) novelty definition and comparison relative to LPO and GFN4Rec, (2) cross-position dependency modeling under the finite-group position-wise surrogate, (3) ranking gain magnitude over the strongest baseline and benchmark generality, and (4) how stochastic diversity advantages translate into practical benefits for deterministic (T=0T=0 greedy) serving, we conducted targeted theoretical and empirical analyses during the rebuttal period and report them in detail below.


1. Response to Weakness 1: Sharper Novelty Definition Relative to LPO and GFN4Rec

We agree that explicitly contrasting GRMO against LPO and GFN4Rec sharpens our contribution, and we will add Table 1 and a dedicated discussion to Sections 2.2-2.3:

MethodProbability-Matching GranularityResolves Cross-Rank Credit Confounding (K=20K=20)?Auxiliary Flow / Critic Networks?Open-Vocabulary Autoregressive LLM Compatibility
LPOSequence-level q(y∣x)q(y\mid x)No (single scalar weight applied to all 20 items)NoneGeneral text; confounds relevant & irrelevant ranks in lists
GFN4RecStepwise discrete item-ID flowPartial (via learned forward/backward flow balance)Yes (learned flow / state-potential networks + candidate enumeration)Designed for closed item-ID graphs; does not natively scale to multi-token open-vocabulary title generation
GRMO (Ours)Shared-parse token-span rank-local mass qi,kq_{i,k}Yes (position-local credit + outer DCG rank weights)None (self-normalized intra-group importance correction)Native end-to-end training on any pretrained LLM
  1. Why sequence-level matching (LPO) falls short for list recommendation: Applying a single sequence target q(y∣x)∝πref(y∣x)exp⁡(βR(x,y))q(y\mid x) \propto \pi_{\mathrm{ref}}(y\mid x)\exp(\beta R(x,y)) across K=20K=20 ordered items amplifies 18–19 irrelevant or seen items whenever a trajectory contains a single Rank-1 hit, while suppressing valid items located inside lower-total-score trajectories. Moreover, finite groups cannot cover the combinatorial ∣I∣20|\mathcal{I}|^{20} list space. GRMO resolves this via shared-parse token-span ownership, length-normalized item log-ratios si,kθs_{i,k}^\theta, and post-standardization outer DCG rank weights wk∝1/log⁡2(k+1)w_k \propto 1/\log_2(k+1). In our fixed-budget controlled ablations, rank-local NDCG-aligned reverse mass-KL outperforms unaligned and forward-KL variants by +0.00072 to +0.00090 NDCG@20, while our unowned-token reference anchor boosts Exact-20 format validity from 51.85% to 98.94%.
  2. Why GFN4Rec does not substitute for GRMO: GFN4Rec parameterizes explicit forward/backward flow functions over discrete item IDs and enumerates valid item sets at each constructive step. In conversational LLM recommendation where each item is generated autoregressively as variable-length natural-language tokens, GRMO requires zero auxiliary flow networks, critics, or partition-function learners, relying solely on reference/proposal log-ratios within sampled groups.

2. Response to Weakness 2: Local Groupwise & Position-Wise Surrogate, Cross-Position Dependencies, and Rank-Band Gains

Thank you for this incisive theoretical observation. While exact population KL minimization over ∣I∣20|\mathcal{I}|^{20} full lists is intractable, GRMO's rank-local surrogate preserves list-wide coherence and improves both head and tail positions for two reasons:

  1. Autoregressive Conditioning Preserves Cross-Position Dependencies: Although group softmax normalization is computed at each rank kk, every item score si,kθ=log⁡πθ(yi,k∣x,yi,<k)πold(yi,k∣x,yi,<k)s_{i,k}^{\theta}=\log\frac{\pi_{\theta}(y_{i,k}\mid x,y_{i,<k})}{\pi_{\mathrm{old}}(y_{i,k}\mid x,y_{i,<k})} is strictly conditioned on the dialog context xx and the entire prefix of previously generated items yi,<ty_{i,<t} (ranks 1…k−11 \dots k-1). Thus, the policy preserves inter-item dependencies such as non-duplication and complementary diversity: under ordinary greedy decoding (repetition_penalty=1.0, zero post-filtering), converged GRMO achieves >98.5% within-list unique titles.
  2. Broad Gain Attribution Across All List Positions: Decomposing Llama-3.2-3B's NDCG@20 gain by rank band (Table 2) demonstrates that GRMO improves not only Rank 1 (+0.00088, 48.5% of total gain), but also middle Ranks 6–10 (+0.00045, 24.7%) and tail Ranks 11–20 (+0.00056, 30.9%):
Recommendation Rank Band0.1283 Baseline ContributionConverged GRMO ContributionNet Band Gain Δ\DeltaShare of Total Gain
Rank 1 (Top-1 item)0.039210.04009+0.0008848.5%
Ranks 2–50.048400.04833-0.00007-4.0%
Ranks 6–10 (Mid-list)0.021910.02236+0.0004524.7%
Ranks 11–20 (Tail-list)0.018780.01934+0.0005630.9%
Total NDCG@200.128300.13012 (0.1301)+0.00182100%

3. Response to Weakness 3: Substantially Larger Gains Upon Full Convergence (+1.42% to +3.71%) and Stratified Generalization

3.1 Full Convergence Surpasses the 1.2% Margin Across Both Backbones (Winning 16/16 Metrics)

The checkpoints reported in our initial submission had not yet fully converged due to limited training epochs prior to the submission deadline (resulting in <1.2% gains). After continuing training to full convergence under the exact same 10,972-row benchmark and T=0T=0 greedy decoding, the converged results are reported below: GRMO significantly outperforms Rank-GRPO (exp∞) across all 16/16 metrics on both backbones, decisively surpassing the 1.2% margin (Table 3):

Backbone / MethodR@5R@10R@15R@20NDCG@5NDCG@10NDCG@15NDCG@20
Qwen2.5-0.5B
Rank-GRPO (exp⁡∞\exp_\infty)0.09460.14680.18130.20470.07440.09150.10160.1079
GRMO (Fully Converged)0.09810.14780.18300.20870.07620.09240.10270.1096
Relative Gain (vs. Paper Table 1)+3.71%+0.67%+0.96%+1.97%+2.45%+0.96%+1.11%+1.60%
Paired Bootstrap pp (vs. Official Dump)p=0.0029+0.93%+0.97%+1.07%p=0.0103+1.10%p=0.052p=0.0340
Llama-3.2-3B
Rank-GRPO (exp⁡∞\exp_\infty)0.11780.17560.21500.23680.09190.11070.12230.1283
GRMO (Fully Converged)0.11870.17740.21760.24190.09280.11190.12360.1301
Relative Gain (vs. Paper Table 1)+0.80%+1.03%+1.19%+2.14%+0.95%+1.09%+1.06%+1.42%
Paired Bootstrap pp (vs. 0.1283 Dump)--p=0.052p=0.0017-p=0.0037p=0.0010p=0.0001
  • Llama-3.2-3B: 0.1301 NDCG@20 (+1.42%, Row 95% CI [+0.00094, +0.00271], p=0.0001p = 0.0001) and 0.2419 Recall@20 (+2.14%, Row 95% CI [+0.00133, +0.00620], p=0.0017p = 0.0017); 3-seed mean: 0.1296 ± 0.0004.
  • Qwen2.5-0.5B: 0.1096 NDCG@20 (+1.60% vs. 0.1079; +1.18% with Row 95% CI [+0.00009, +0.00247], p=0.0340p = 0.0340 even vs. the re-evaluated official checkpoint 0.10834), 0.2087 Recall@20 (+1.97%), and +3.71% Recall@5 (p=0.0029p = 0.0029); 3-seed mean: 0.1092 ± 0.0004.

3.2 Benchmark Scope & Stratified Generalization Across Recommendation Regimes

Following Rank-GRPO, our work targets multi-turn natural-language conversational list recommendation, where Reddit-v2 is currently the only public large-scale benchmark combining multi-turn natural-language requests (382,829 train pairs; 10,972 test rows) with genuine one-to-many ground-truth annotations (unlike item-ID click logs in MovieLens/Amazon). Beyond validating across two distinct model families spanning a 6$\times$ parameter scale, we evaluated generalization across five recommendation strata varying in intent multiplicity (∣GT∣|GT|) and seen-history sparsity (Table 4, Llama-3.2-3B):

Recommendation StratumRowsBaseline NDCG@20Converged GRMO NDCG@20Δ\DeltaNDCG@20Baseline Recall@20Converged GRMO Recall@20Δ\DeltaRecall@20
Single-target intent ($GT=1$)7,1820.117020.11899+0.001970.24798
Moderate multi-intent ($2 \leGT\le 4$)2,6230.142410.14408+0.001670.23504
Broad multi-intent ($GT\ge 5$)1,1670.165970.16716+0.001190.18446
Cold-history dialogs (seen = 0)4,0020.119380.12209+0.00271 (+2.27%)0.221700.22672+0.00502 (+2.26%)
Warm-history dialogs (seen >= 1)6,9700.133420.13472+0.00131 (+0.98%)0.247570.25057+0.00301 (+1.21%)

GRMO achieves positive NDCG@20 and Recall@20 gains across 100% (5/5) of strata, including +2.27% NDCG@20 on cold-start conversations (seen = 0). We will also explicitly note cross-domain conversational extensions in our Limitations section.


4. Response to Weakness 4: Translation of Diversity Benefits to Deterministic (T=0T=0 Greedy) Serving

This is an excellent practical question. To directly measure how GRMO benefits production systems that serve purely deterministic (T=0T=0 greedy) recommendations, we evaluated population-level catalog coverage, effective catalog size exp⁡(H)\exp(H), Gini popularity concentration, and unique ground-truth item recovery directly on the deterministic T=0T=0 greedy predictions across all 2,050 unique test queries on both backbones (Table 5):

Deterministic (T=0T=0 Greedy) Serving MetricQwen: Official Rank-GRPO (exp∞)Qwen: Converged GRMOLlama: 0.1283 BaselineLlama: Converged GRMO
Catalog-valid unique items @5 (↑\uparrow)1,6601,776 (+116)1,9892,027 (+38)
Catalog-valid unique items @10 (↑\uparrow)2,3622,560 (+198)2,8842,931 (+47)
Catalog-valid unique items @20 (↑\uparrow)3,3013,473 (+172)3,8603,985 (+125)
Effective catalog size exp⁡(H)\exp(H) @20 (↑\uparrow)1,283.81,436.2 (+11.87%)1,578.71,644.6 (+4.18%)
Gini concentration index @20 (↓\downarrow)0.70670.6927 (-0.0140)0.68300.6809 (-0.0021)
Between-query Jaccard distance @20 (↑\uparrow)0.98490.98700.98850.9887
Distinct ground-truth items recovered @20 (↑\uparrow)1,2671,331 (+64 movies)1,5671,593 (+26 movies)

Why training-time mass matching translates directly to T=0T=0 greedy serving: Under sparse binary feedback, standard advantage maximization pushes the argmax across many distinct user contexts toward a narrow set of globally popular blockbusters. Because GRMO matches reference-anchored intra-group probability mass during training, head blockbusters are prevented from cannibalizing context-specific niche items. Consequently, even under purely deterministic T=0T=0 greedy decoding, distinct user queries peak on distinct context-appropriate items, expanding effective catalog utilization by +11.87% (Qwen) and +4.18% (Llama), lowering Gini concentration, and recovering +64 (Qwen) and +26 (Llama) additional unique ground-truth movies. We will add Table 5 to Section 4.2.

Thank you again for your constructive review; we hope these theoretical clarifications and new empirical findings address all of your concerns.

Reviewer_tvmBOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper focuses on reinforcement learning optimization for generative recommendation and proposes Group Reward-Mass Optimization (GRMO), which constructs a reward-guided target probability distribution over sampled candidate groups and optimizes the recommendation policy through groupwise probability-mass matching. The authors conduct experiments on the Reddit-v2 dataset and report performance improvements over GRPO and Rank-GRPO. However, the current work still has several significant limitations. First, the algorithmic novelty is limited, and the distinctions from existing reward-distribution matching, groupwise policy optimization, and KL-regularized RL methods are not sufficiently clarified. Second, the paper lacks direct empirical validation of its core motivation that sparse feedback leads to probability concentration and limits access to high-quality alternative recommendations. Third, important experimental and implementation details are insufficiently described, which affects reproducibility. The paper also lacks systematic ablation studies of its key components. In addition, the experiments are conducted on only a single dataset and a single recommendation setting, making it difficult to sufficiently support the generality and practical effectiveness of the proposed method. Therefore, I do not recommend accepting this paper in its current form and suggest that the authors carefully revise the manuscript by addressing the above concerns.

Critical strengths
  1. The authors propose a new framework for generative recommendation.

  2. The proposed method is evaluated on one dataset.

Critical weaknesses
  1. The algorithmic novelty of the paper is quite limited. GRMO essentially constructs a reward-based target distribution over sampled candidates and updates the policy through distribution matching, which is highly similar in overall formulation to existing reward-distribution matching, groupwise policy optimization, and KL-regularized RL methods. However, the paper does not sufficiently analyze the substantive differences between GRMO and these existing approaches in the related work section. It also lacks a clear motivation explaining why group-level probability-mass matching is particularly necessary for generative recommendation and what specific limitations of existing objectives cannot be addressed by prior methods. Therefore, it is difficult to determine whether GRMO introduces a genuinely new optimization mechanism or mainly transfers existing distribution-matching ideas to the generative recommendation setting.

  2. The paper lacks direct empirical validation of its core motivation. For example, the authors state in the abstract and introduction that “under sparse feedback, generation probability may concentrate on a narrow set of outputs, potentially limiting access to alternative high-quality recommendations,” and use this as an important motivation for introducing GRMO. However, the paper does not provide sufficiently detailed experimental evidence to demonstrate that this phenomenon actually exists and is significant.

  3. The paper also omits many important experimental and implementation details. For example, it is unclear how the SFT training pairs are constructed, how the reward function is designed in practice, how the dataset is processed, and what the exact model inputs and outputs are. The proposed approach also does not appear to follow the Semantic ID-based generative recommendation paradigm, but the paper does not clearly explain how the recommendation lists are generated. Moreover, the authors do not provide reproducible code, making it difficult to verify or reproduce the reported results.

  4. The paper does not provide ablation studies. Without systematic ablations, it is difficult to verify whether the proposed components actually contribute to the performance improvements or to identify the true source of the reported gains.

  5. The experiments are conducted on only a single dataset. The lack of evaluation on additional recommendation datasets, different recommendation settings, or different application domains makes the current empirical evidence insufficient to support the generality of the proposed method. As a result, the reported performance improvements are not sufficiently convincing.

Questions for authors

Please see above weakness.

5 comments

Thank you for taking the time to review our paper and share your feedback. To address the five core concerns raised in your review: (1) algorithmic novelty vs. prior distribution-matching and groupwise RL methods, (2) direct empirical validation of our core motivation, (3) generative recommendation paradigm (Semantic IDs vs. end-to-end textual generation), model inputs/outputs, and reward design, (4) systematic ablation studies, and (5) benchmark scope and multi-dimensional generalization, we provide detailed point-by-point clarifications below, supported by the exact equations in our paper and extensive new experiments conducted during the rebuttal period.


1. Algorithmic Novelty and Substantive Differences from Prior Methods

Sections 2.2 and 2.3 discuss RAML, MPO, LLM distribution matching, GFN4Rec, FlowRL, BRAIn, and LPO. Rather than simply transferring generic distribution matching to recommendation, GRMO introduces three optimization mechanisms specifically necessitated by autoregressive ranked list generation (K=20K=20 items):

  1. Substantive difference from Groupwise Policy Optimization (GRPO / Rank-GRPO): GRPO and Rank-GRPO update each sampled candidate independently through a clipped policy ratio −min⁡(ρi,kAi,k,clip(ρi,k)Ai,k)-\min(\rho_{i,k} A_{i,k}, \mathrm{clip}(\rho_{i,k}) A_{i,k}), imposing no coupled constraint on how probability mass is partitioned across candidates. Under sparse hits, this concentrates probability onto a few dominant head items. GRMO normalizes both policy mass p⋅,kp_{\cdot,k} and importance-corrected reward-tilted target mass qi,k=softmaxi(βAi,k+si,kref)q_{i,k} = \mathrm{softmax}_i(\beta A_{i,k} + s_{i,k}^{\mathrm{ref}}) within each group and minimizes KL(p⋅,k∥q⋅,k)\mathrm{KL}(p_{\cdot,k} \Vert q_{\cdot,k}). Once an item's policy share pi,kp_{i,k} reaches its target share qi,kq_{i,k}, the gradient vanishes and automatically prevents further cannibalization of other high-quality candidates.
  2. Substantive difference from Sequence-Level Distribution Matching (RAML / MPO / BRAIn / LPO): Assigning a single scalar sequence weight to a 20-item recommendation list causes severe cross-rank credit confounding (amplifying 18–19 irrelevant positions in a trajectory that happens to contain one head hit, while suppressing valid items inside lower-total-score lists) and suffers from ∣I∣20|\mathcal{I}|^{20} combinatorial sparsity. GRMO introduces shared-parse token-span ownership for rank-local empirical mass matching, length-invariant item log-ratios si,kθs_{i,k}^\theta, and post-standardization outer DCG rank weights wk∝1/log⁡2(k+1)w_k \propto 1/\log_2(k+1).
  3. Substantive difference from GFN4Rec: Unlike GFN4Rec, which requires parameterizing and training forward/backward flow networks and enumerating discrete candidate sets at every step, GRMO requires no auxiliary flow network, value critic, or partition-function learner, optimizing open-vocabulary pretrained LLMs end to end.

2. Direct Empirical Validation of Our Core Motivation

Both Section 4.2 (Table 2, Figures 2–3) and our new head-to-head deterministic (T=0T=0) and converged evaluations directly demonstrate that sparse-reward RL concentrates probability onto narrow popular items and that GRMO substantially improves both ranking accuracy and catalog/path accessibility:

2.1 Direct Head-to-Head Evidence Under Deterministic Greedy Decoding (T=0T=0, All 10,972 Test Rows)

Comparing fully converged GRMO directly against official Rank-GRPO (exp∞) under purely deterministic T=0T=0 decoding across all 2,050 unique test queries (Table 1), Rank-GRPO (exp∞) exhibits higher popularity concentration (higher Gini index, narrower catalog coverage). In contrast, GRMO outperforms Rank-GRPO (exp∞) on all 16/16 ranking metrics across both backbones while substantially expanding effective catalog coverage and recovering dozens of additional unique long-tail ground-truth movies:

Evaluation Metric (10,972 Rows, T=0T=0 Greedy)Qwen: Official Rank-GRPO (exp∞)Qwen: Converged GRMOLlama: Rank-GRPO (exp∞) / BaseLlama: Converged GRMO
NDCG@20 (↑\uparrow)0.1079 (re-eval 0.1083)0.1096 (+1.60%, p=0.034p=0.034)0.12830.1301 (+1.42%, p=0.0001p=0.0001)
Recall@20 (↑\uparrow)0.2047 (re-eval 0.2065)0.2087 (+1.97%)0.2368 (base 0.2381)0.2419 (+2.14%, p=0.0017p=0.0017)
Recall@5 (↑\uparrow)0.09460.0981 (+3.71%, p=0.0029p=0.0029)0.1178 (base 0.1183)0.1187 (+0.80%)
Catalog-valid unique items @20 (↑\uparrow)3,301 items3,473 items (+172 items)3,860 items3,985 items (+125 items)
Effective catalog size exp⁡(H)\exp(H) @20 (↑\uparrow)1,283.81,436.2 (+11.87%)1,578.71,644.6 (+4.18%)
Gini popularity concentration @20 (↓\downarrow)0.70670.6927 (less concentrated)0.68300.6809 (less concentrated)
Distinct ground-truth items hit @20 (↑\uparrow)1,267 movies1,331 movies (+64 movies)1,567 movies1,593 movies (+26 movies)

2.2 Direct Stochastic Trajectory Evidence (65,536 Paired Trajectories Across 512 Queries)

Under matched sampling budgets (Section 4.2), GRMO increases the high-quality trajectory survival rate (NDCG@20≥0.10\mathrm{NDCG@20} \ge 0.10) from 36.11% to 42.18% (+16.78% relative) on Llama-3.2-3B and from 31.78% to 33.57% (+5.63% relative) on Qwen2.5-0.5B, while expanding the Query Separation Margin (Between-query minus Within-query Hellinger distance) by +22.54% (Llama) and +11.50% (Qwen) (10,000-resample paired 95% CIs strictly exclude zero). Against official Rank-GRPO (exp∞) at T=1.3T=1.3, GRMO also improves Oracle Best-of-8 discovery (0.1707 vs. 0.1650, +0.0057).


3. Generative Paradigm (Semantic IDs vs. Textual Generation), I/O, SFT Construction, and Reward Design

All task definitions and mathematical formulations are specified in the paper, and we summarize them clearly below (and will further expand step-by-step pseudocode in the revised Appendix):

  1. Why Semantic IDs Are Not Used: Generative recommendation comprises two established paradigms: (a) discrete Semantic-ID sequence indexing (e.g., TIGER) for implicit ID logs, and (b) end-to-end natural-language item list generation (e.g., BIGRec and our direct baseline Rank-GRPO). As explicitly defined in Section 3.1 (Eq. 1) and Section 3.2, our paper follows the exact same end-to-end conversational recommendation paradigm as Rank-GRPO: Rank-GRPO does not use Semantic IDs either. Retaining the identical input-output space and generation paradigm is essential for a fair comparison of optimization objectives.
  2. Exact Model Inputs/Outputs, Dataset Processing, & SFT Pairs (Sections 3.1-3.2):
    • Input xx: Multi-turn natural-language Reddit conversation history expressing user preferences and previously watched movies;
    • Output yy: An autoregressively generated ranked text list of K=20K=20 catalog movie titles with release years (one item per line);
    • Dataset & SFT: Following Rank-GRPO's standard Reddit-v2 preprocessing pipeline, training uses 382,829 length-filtered dialog-list pairs (72,233 unique prompts) and evaluation uses 10,972 rows (2,050 unique prompts). SFT fine-tunes the base LLM via standard autoregressive cross-entropy on ground-truth recommendation lists.
  3. Practical Reward Function & Complete Loss (Section 3.4 Eqs. 15-21 & Appendix B Eqs. 35-38):
    • A single shared parse extracts (normalized_title, year) and exact character/token spans per rank;
    • Movies mentioned in the user's conversation history (seen items) are filtered out to prevent reward hacking via history repetition;
    • Unseen items matching a ground-truth title within ±2\pm 2 release years receive position-preserved binary credit ri,k∈{0,1}r_{i,k} \in \{0, 1\} (averaged across annotations for multi-annotation prompts), without shifting rewards across invalid or unmatched positions;
    • The total loss LGRMO=Lmass+LlengthL_{\mathrm{GRMO}} = L_{\mathrm{mass}} + L_{\mathrm{length}} combines token-normalized rank-local reverse mass-KL with an asymmetric-clipped ([0.94,1.08][0.94, 1.08]) list-length auxiliary penalty on early stopping and overflow (Eq. 21 & Appendix B Eqs. 35-38).

4. Systematic Ablation Studies

Holding initialization, data order, and per-step rollout budget (B×GB \times G) strictly fixed across each paired comparison, we conducted controlled single-variable ablations across five key design axes on the full 10,972-row benchmark. Table 2 summarizes the relative performance gains brought by each component:

Ablated ComponentControlled Single-Variable Comparison (All Other Settings & Budget Fixed)Relative NDCG@20 GainKey Secondary / Format ImpactEmpirical Finding
1. Core ObjectiveGRMO (groupwise mass matching) vs. Rank-GRPO (exp∞) (Qwen / Llama)+1.60% / +1.42%Recall@20 +1.97% / +2.14%GRMO wins all 16/16 metrics (p=0.034p = 0.034 / p=0.0001p = 0.0001)
2. Outer DCG Rank Weight+ Outer DCG wkw_k vs. Uniform rank weight (Control)+0.89%EOS-only ratio tweak gives +0.11%Restores top-rank priority after within-rank standardization
3. Mass-KL DirectionNDCG-Aligned Reverse KL vs. Forward KL+0.69%Recall@5 +2.24% (+0.46% vs. unaligned Rev-KL)Prevents diffusing probability mass onto zero-reward tails
4. Credit ScopePosition-Local Credit vs. Suffix reward-to-go credit+0.96%Preserves head-rank credit accuracyPrevents misattributing tail hits to upstream misses
5. Unowned-Token AnchorEnabled Unowned-Token Anchor vs. Disabled+0.72%+47.1 pp Exact-20 validity (+90.8% relative)Eliminates unparsed format drift, sustaining >96.6% Exact-20 validity

All controlled ablation tables will be incorporated into Section 4.3 of the revised manuscript.


Reviewer_KqMVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Group Reward-Mass Optimization (GRMO) for generative conversational recommendation. GRMO constructs a reward-tilted target distribution within each sampled candidate group and optimizes the allocation of probability mass by minimizing reverse KL divergence. The implementation uses a position-level surrogate objective, together with annotation-averaged rewards, target-KL calibration, and an auxiliary length loss. Experiments on Reddit-v2 with two model backbones report improvements in ranking metrics and better access to high-quality recommendation paths under a fixed sampling budget.

Critical strengths

1.The research objective addresses a practical need in generative recommendation. A single user intent can admit several reasonable recommendations. The paper considers both ranking quality and access to high-quality alternatives, rather than evaluating only a single generated list.

2.The empirical analysis examines several relevant aspects of quality and accessibility. The paper reports ranking improvements in the same direction on both backbones and further investigates sampling temperature, differences across query contexts, and the discovery of high-quality paths under a fixed budget. The authors also explain the limitations of the quality-adjusted coverage metric, allowing these results to be interpreted within an appropriate scope.

Critical weaknesses

1.The contribution beyond existing methods and the attribution of the gains remain insufficiently established. The related-work discussion explicitly acknowledges the connections to BRAIn and LPO, but the experiments do not directly test what the recommendation-specific instantiation adds to these distribution-matching mechanisms. The paper also does not specify, component by component, which reward-processing steps, position weights, and auxiliary losses are shared with Rank-GRPO. Nor does it provide a controlled comparison in which only the core optimization objective changes while the remaining settings and training budget are held fixed. I am wondering how much of the improvement is attributable to the GRMO objective itself.

2.There are gaps in the reliability and reproducibility of the key experimental findings. Relative to Rank-GRPO (exp∞), the absolute NDCG@20 gains are only 0.0007 and 0.0006 on the two backbones. Although some diversity analyses include uncertainty estimates, Table 1 reports neither confidence intervals for these main ranking differences nor variation across training seeds. In addition, the path-coverage analysis depends on path clusters c_{q,s}, whose construction is not specified. Table 2 and Figures 2–3 also do not clearly identify results for the strongest Rank-GRPO variant. These omissions make it difficult to assess the stability of the small ranking advantage and whether the gains in high-quality path accessibility hold relative to the strongest baseline.

Questions for authors

1.Which reward-processing steps, position weights, and auxiliary length losses are shared between GRMO and Rank-GRPO? Is there a comparison that changes only the core optimization objective while holding these components, initialization, and training budget fixed?

2.Could you provide paired confidence intervals for the main ranking differences in Table 1, together with results across multiple training seeds? Please also specify which variant is labeled “GRPO” in Table 2 and Figures 2–3, and clarify whether similar gains in path accessibility are observed relative to Rank-GRPO (exp∞).

3.How are the path clusters c_{q,s} constructed, or how is equivalence between paths determined? Are there thresholds or other parameters that affect the results?

3 comments

We sincerely thank you for your thoughtful, detailed, and constructive review.


1. Response to Weakness 1 & Question 1: Shared Components with Rank-GRPO, Distinction from BRAIn/LPO, and Controlled Ablations

1.1 Component-by-Component Specification vs. Rank-GRPO (exp∞)

To eliminate any ambiguity in gain attribution, Table 1 lists every module shared between Rank-GRPO (exp∞) and GRMO alongside the sole core optimization difference:

Module / ComponentRank-GRPO (exp∞)GRMO (Ours)Shared?
Backbone & I/O ProtocolSame SFT initialization; multi-turn dialog x→20x \rightarrow 20-item ranked movie listIdenticalYes
Raw Position Reward ri,kr_{i,k}Position-preserving binary hit (normalized title + ±2\pm 2-year tolerance, seen-item filtering, multi-annotation mean)Identical (Sec. 3.4)Yes
Within-Rank CenteringMean-centering / std scaling across GG sampled siblings at rank kkSame position-preserved centering structureYes
Length Auxiliary Loss LlengthL_{\mathrm{length}}Penalizes early stopping (<20<20 items) and list overflow (>20>20 items)Same K=20K=20 target-length constraintYes
Core Policy Objective (Sole Core Difference)Uncoupled Clipped Ratio Surrogate: −min⁡(ρi,kAi,k,clip(ρi,k)Ai,k)-\min(\rho_{i,k} A_{i,k}, \mathrm{clip}(\rho_{i,k}) A_{i,k})Coupled Groupwise Mass-KL Matching: KL(p⋅,k∥q⋅,k)\mathrm{KL}(p_{\cdot,k} \Vert q_{\cdot,k}) with qi,k=softmaxi(βAi,k+si,kref)q_{i,k} = \mathrm{softmax}_i(\beta A_{i,k} + s_{i,k}^{\mathrm{ref}})Core Novelty

What GRMO's recommendation-specific instantiation adds beyond BRAIn and LPO: BRAIn and LPO perform sequence-level distribution matching over entire responses. In ranked list generation (K=20K=20 ordered items), sequence-level matching suffers from two structural limitations:

  1. Cross-rank credit confounding: a scalar sequence reward assigns identical positive target weight to all 20 positions of a trajectory (even if 18-19 positions miss or contain noise), while suppressing valid items at individual positions inside lower-total-reward trajectories;
  2. Combinatorial support sparsity: within the ∣I∣20|\mathcal{I}|^{20} combinatorial space, a finite group size GG cannot form dense support at the full-sequence level.

GRMO introduces shared-parse token-span ownership to decouple distribution matching into each recommendation position kk, incorporating length-normalized item log-ratios si,kθs_{i,k}^\theta (removing multi-token title-length bias) and post-standardization outer DCG rank weights wk∝1/log⁡2(k+1)w_k \propto 1/\log_2(k+1). Crucially, unlike Rank-GRPO, which amplifies positive advantages independently per sample, GRMO couples candidate probability shares via intra-group softmax: once a high-reward item's current policy mass pi,kp_{i,k} reaches or exceeds its target mass qi,kq_{i,k}, ∇KL(p⋅,k∥q⋅,k)\nabla \mathrm{KL}(p_{\cdot,k} \Vert q_{\cdot,k}) automatically curbs further probability concentration, preserving probability space for alternative valid items.

1.2 Controlled Single-Variable Ablations (Fixed Initialization, Data Order, & Budget)

Holding initialization, data order, per-step rollout budget (B×GB \times G), and greedy evaluation strictly fixed across each paired comparison, we conducted single-variable controlled ablations on the core objective and each recommendation-specific component. Table 2 summarizes the relative performance gains brought by each module:

Ablated ModuleControlled Single-Variable Comparison (Initialization, Other Components, & Budget Fixed)Relative NDCG@20 GainKey Secondary / Format EffectAttribution Takeaway
1. Core Policy ObjectiveGRMO (groupwise mass matching) vs. Rank-GRPO (exp∞) (clipped ratio)+1.60% / +1.42%Recall@20 +1.97% / +2.14%Replacing only the core objective wins 16/16 metrics on Qwen / Llama
2. Outer DCG Rank Weight+ Outer DCG Weighting wkw_k vs. Uniform rank weight (Control)+0.89%EOS-only ratio tweak gives +0.11%Confirms the key role of DCG rank priority and rules out length-loss confounding
3. KL Direction & AlignmentNDCG-Aligned Reverse Mass-KL vs. Forward Mass-KL+0.69%Recall@5 +2.24% (+0.46% vs. unaligned Rev-KL)Prevents mean-seeking Forward KL from diffusing mass onto zero-reward tails
4. Credit Assignment ScopePosition-Local Credit vs. Suffix reward-to-go credit+0.96%Preserves top-rank precisionSuffix credit misattributes downstream hits to upstream misses
5. Format Span Anchor+ Unowned-Token Reference Anchor vs. Without anchor+0.72%+47.1 pp Exact-20 rate (+90.8% relative)Eliminates unparsed format drift, sustaining >96.6% Exact-20 validity

2. Response to Weakness 2 & Question 2: Converged Gains, Paired Confidence Intervals, Multi-Seed Results, and "GRPO" Identity Clarification

2.1 Main Ranking Comparison After Full Convergence and Paired Bootstrap 95% CIs

Due to limited training steps prior to the submission deadline, the checkpoints in our initial submission had not yet fully converged. After continuing training to full convergence and evaluating on the exact same 10,972 evaluation rows under deterministic greedy decoding (T=0T=0, with no post-processing), the converged results are shown in Table 3: GRMO outperforms Rank-GRPO (exp∞) across all 16/16 metrics (8/8 on each backbone), expanding the NDCG@20 gain to 2.5-3.0$\times$ that of the initial submission:

Backbone / MethodR@5R@10R@15R@20NDCG@5NDCG@10NDCG@15NDCG@20
Qwen2.5-0.5B
Rank-GRPO (exp⁡∞\exp_\infty) (Paper Table 1)0.09460.14680.18130.20470.07440.09150.10160.1079
Rank-GRPO (exp⁡∞\exp_\infty) (Official weight re-eval)0.09460.14640.18130.20650.07440.09140.10150.1083
GRMO (under-converged)0.09470.14660.18160.20700.07460.09160.10180.1086
GRMO (Fully Converged)0.09810.14780.18300.20870.07620.09240.10270.1096
Gain vs. Rank-GRPO (exp⁡∞\exp_\infty)+3.71%+0.67%+0.96%+1.97%+2.45%+0.96%+1.11%+1.60% (+0.0017)
Llama-3.2-3B
Rank-GRPO (exp⁡∞\exp_\infty) (Paper Table 1)0.11780.17560.21500.23680.09190.11070.12230.1283
GRMO (under-converged)0.11860.17580.21570.23860.09250.11120.12280.1290
GRMO (Fully Converged, twice reload-verified)0.11870.17740.21760.24190.09280.11190.12360.1301
Gain vs. Rank-GRPO (exp⁡∞\exp_\infty)+0.80%+1.03%+1.19%+2.14%+0.95%+1.09%+1.06%+1.42% (+0.0018)

Paired Bootstrap 95% Confidence Intervals (10,000 resamples, both row-level N=10,972N=10,972 and query-cluster level N=2,050N=2,050):

  • Llama-3.2-3B (paired against matched 0.12830 baseline dump):
    • ΔNDCG@20=+0.00182\Delta\mathrm{NDCG@20} = +0.00182, Row 95% CI [+0.00094, +0.00271] (p=0.0001p = 0.0001), Query-Cluster 95% CI [+0.00084, +0.00283] (p=0.0005p = 0.0005);
    • ΔRecall@20=+0.00374\Delta\mathrm{Recall@20} = +0.00374, Row 95% CI [+0.00133, +0.00620] (p=0.0017p = 0.0017), Query-Cluster 95% CI [+0.00104, +0.00648] (p=0.0067p = 0.0067);
    • ΔNDCG@15=+0.00141\Delta\mathrm{NDCG@15} = +0.00141 (Row 95% CI [+0.00055, +0.00227], p=0.0010p = 0.0010), ΔNDCG@10=+0.00126\Delta\mathrm{NDCG@10} = +0.00126 (Row 95% CI [+0.00045, +0.00212], p=0.0037p = 0.0037).
  • Qwen2.5-0.5B (directly paired row-by-row against the re-evaluated official Rank-GRPO (exp∞) dump 0.10834):
    • Even against the higher re-evaluated official Rank-GRPO (exp∞) score (0.10834), GRMO (0.10963) achieves ΔNDCG@20=+0.00128\Delta\mathrm{NDCG@20} = +0.00128, Row 95% CI [+0.00009, +0.00247] (p=0.0340p = 0.0340), Query-Cluster 95% CI [+0.00006, +0.00253] (p=0.0414p = 0.0414), with substantial head improvements: ΔRecall@5=+0.00351\Delta\mathrm{Recall@5} = +0.00351 (+3.71%, Row 95% CI [+0.00129, +0.00575], p=0.0029p = 0.0029) and ΔNDCG@5=+0.00182\Delta\mathrm{NDCG@5} = +0.00182 (+2.44%, Row 95% CI [+0.00042, +0.00320], p=0.0103p = 0.0103).

Variation Across Multiple Training Seeds:

  • Llama-3.2-3B (3 independent seeds): NDCG@20 = 0.1301, 0.1294, 0.1293 (0.1296 ± 0.0004, every seed strictly exceeding Rank-GRPO (exp∞) at 0.1283).
  • Qwen2.5-0.5B (3 independent runs/seeds): NDCG@20 = 0.1096, 0.1093, 0.1088 (0.1092 ± 0.0004, every seed strictly exceeding Rank-GRPO (exp∞) at 0.1079 / 0.1083). We will include full paired 95% CIs and multi-seed statistics in Table 1.

2.2 Identity of "GRPO" in Table 2 / Figures 2-3 and Direct Comparison to Rank-GRPO (exp∞)

  • Clarification: Thank you for flagging this detail. In Table 2 and Figures 2-3 (65,536 trajectories across 512 fixed queries), "GRPO" denotes the designated initial RL baseline checkpoint for each backbone rather than the final Rank-GRPO (exp∞) checkpoint. We will specify exact variant names in all revised table headers and figure captions.
  • Direct Head-to-Head Accessibility & Coverage vs. Official Rank-GRPO (exp∞):
    1. Stochastic Path Accessibility (T∈{0.7,1.0,1.3}T \in \{0.7, 1.0, 1.3\}): Directly comparing against official Rank-GRPO (exp∞) across 2,048 trajectories per temperature, GRMO preserves or exceeds high-reward effective paths at T=1.0T=1.0 (2.514 vs. 2.503) and T=1.3T=1.3 (3.997 vs. 3.981), while achieving higher mean reward (0.0999 vs. 0.0991), higher high-reward title coverage (+0.0218), and higher Oracle Best-of-8 discovery (0.1707 vs. 0.1650, +0.0057) at T=1.3T=1.3.
    2. Deterministic (T=0T=0) Catalog Coverage vs. Official Rank-GRPO (exp∞): Across all 2,050 test queries under T=0T=0 greedy decoding on Qwen2.5-0.5B, converged GRMO expands catalog-valid unique items (@20) from 3,301 (Rank-GRPO (exp∞)) to 3,473 (+172 items), increases effective catalog size exp⁡(H)\exp(H) from 1,283.8 to 1,436.2 (+11.87%), lowers Gini popularity concentration (0.7067 →\rightarrow 0.6927), and recovers +64 additional unique ground-truth movies (1,267 →\rightarrow 1,331). This confirms that GRMO achieves substantial advantages over the strongest Rank-GRPO (exp∞) baseline in both ranking accuracy and alternative item coverage.

3. Response to Question 3: Exact Construction of Path Clusters cq,sc_{q,s} and Parameter Sensitivity

We will add the complete formal specification of cq,sc_{q,s} to Section 4.2 and the Appendix:

  1. Canonical Path Representation: Each sampled trajectory ss for query qq is parsed into its ordered top-dd (d=5d=5, with d∈{10,20}d \in \{10, 20\} verified for sensitivity) catalog-valid, unseen (normalized_title, release_year) pairs. Lists with fewer than dd valid items are padded with a shared <MISSING> placeholder so hallucinated or unparsed strings never create spurious clusters.
  2. DCG-Weighted Prefix-Overlap Distance: Let Ar,BrA_r, B_r denote the valid item sets in the top rr ranks of paths a,ba, b. We define: S(a,b)=∑r=1dwr∣Ar∩Br∣r∑r=1dwr,wr=1log⁡2(r+1),dist(a,b)=1−S(a,b).S(a, b) = \frac{\sum_{r=1}^{d} w_r \frac{|A_r \cap B_r|}{r}}{\sum_{r=1}^{d} w_r}, \quad w_r = \frac{1}{\log_2(r+1)}, \qquad \mathrm{dist}(a, b) = 1 - S(a, b).
  3. Reward-Blind Pooled Clustering: For each query qq, trajectories from both baseline and GRMO (32+32=6432 + 32 = 64 paths) are pooled and clustered without access to rewards via average-linkage hierarchical clustering at prefix-overlap threshold τsim=0.60\tau_{\mathrm{sim}} = 0.60 (distance cutoff 0.40, deterministic lexicographic tie-breaking) to yield shared cluster labels cq,sc_{q,s}, from which DHQ(τ)=mτMexp⁡(−∑cpc(τ)log⁡pc(τ))D_{\mathrm{HQ}}(\tau) = \frac{m_\tau}{M} \exp(-\sum_c p_c^{(\tau)} \log p_c^{(\tau)}) is computed.
  4. Sensitivity & Clustering-Free Verification:
    • Conclusions remain consistent across reward thresholds τ∈{0.05,0.10,0.15,0.20,0.25}\tau \in \{0.05, 0.10, 0.15, 0.20, 0.25\} and temperatures T∈{0.7,1.0,1.3}T \in \{0.7, 1.0, 1.3\} (for example, on Llama-3.2-3B, 95% CIs for ΔDHQ(τ)\Delta D_{\mathrm{HQ}}(\tau) strictly exclude zero across the entire τ∈[0.10,0.25]\tau \in [0.10, 0.25] interval).
    • Furthermore, parameter-free and clustering-free metrics, including rank-discounted Hellinger distances (+11.50% / +22.54% Query Separation Margin on Qwen/Llama, with 95% CIs excluding zero), pairwise Top-20 Jaccard distance, and T=0T=0 effective catalog size exp⁡(H)\exp(H) (+11.87% on Qwen, +4.18% on Llama), all independently confirm the same conclusion.

Thank you again for your thorough review. We believe these controlled ablations, expanded converged ranking gains, paired confidence intervals, and formal clustering definitions fully address your concerns, and we respectfully ask for your positive consideration in the final assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.