acceptodds
Under review as a conference paper at ICLR 2027

Rethinking Exploration in Generative Recommendation: From More Candidates to Better Candidate Sets

Abstract

Autoregressive generative recommendation uses reinforcement learning (RL) to align sequence generation with recommendation utility. Candidate groups determine experiences for policy learning and choices at inference. However, multi-candidate generation faces a structural exploration–quality trade-off: expanding the search space does not necessarily improve the candidate group. This manifests in two failures. (1) Candidate homogenization: probability-driven decoding repeatedly expands shared high-probability prefixes, spending budget on similar trajectories while leaving alternative decisions underexplored. (2) Candidate-quality degradation: stochastic exploration can discover a few high-reward outcomes while introducing many low-quality candidates, increasing Reward Max but decreasing Reward Mean. Candidate count and maximum reward alone do not characterize effective exploration. These failures motivate learning complementary exploration while controlling quality costs. We propose Q-ROLE (Quality-Gated Role Exploration), a lightweight framework with two coupled components. Role-conditioned generation introduces persistent Anchor and Explorer identities within a shared policy, making task-focused exploitation and complementary exploration learnable responsibilities. A quality-gated coverage reward grants a bonus for divergence from the Anchor only when an Explorer meets an Anchor-relative base-reward threshold; otherwise, it receives only its base reward. The learned roles are retained during inference, carrying this division of responsibilities into candidate generation. Q-ROLE preserves the model backbone and requires no auxiliary prefix-value model. Extensive experiments on public recommendation benchmarks demonstrate that Q-ROLE improves a range of strong recommendation baselines, delivering higher recommendation accuracy under matched candidate budgets, while online experiments in an industrial generative reranking system report improved ranking performance.

Primary area:
reinforcement learning
Keywords:
generative recommendation, reinforcement learning, role-conditioned generation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_WFjvOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper presnet Q-ROLE, which aims to address the challenge of effective exploration in auto-regressive generative recommendation. In particular, Q-ROLE is motivated by two trade-offs: 1) probability-driven decoding spends more on high-probability prefixes; 2) stochastic exploration suffers from low-quality results. To resolve them, Q-ROLE introduces quality-gated exploration, which learnable role embeddings are introduces to provide signals about exploitation and exploration, respectively. In experiment part, authros conduct offline evaluation on Amazon with RL basliens and HR evaluation. Moreover, online ab tests results on industrial system are also provides.

Critical strengths

Lightweight Design: Rather than introducing additional value networks or prefix tree search at inference, Q-ROLE embeds exploration responsibilities directly into learnable soft role tokens. This provide an efficient approach to achieve exploration.

Critical weaknesses

Incremental and well-known motivations

Section 3 frames two "insights" as central motivational findings, but both represent well-documented properties in sequence generation and RL:

a) Candidate count != coverage: Prefix collapse and early trajectory redundancy under likelihood-driven decoding (e.g., beam search) are known limitations in autoregressive models operating on long-tail item distributions;

b) Exploration brings noise: Stochastic temperature sampling naturally increases variance—discovering higher maximum rewards at the expense of adding low-utility candidates that degrade the average.

Missing simple decoding baseline

The paper argues that standard beam search collapses onto shared prefixes and random sampling introduces low-quality noise. However, the experiments compare Q-ROLE primarily against standard Beam Search, unconstrained Random Sampling, APAO, and V-STAR. Crucially, the authors do not evaluate simpler, training-free decoding heuristics that directly target prefix collapse, such as:

a) Prefix budget capping: Enforcing a maximum budget on the number of candidates permitted to share the same top-kk prefix during beam expansion (e.g., capping any single prefix at 2 slots);

b) Modern truncation sampling: Evaluating well-tuned sampling strategies like Nucleus (Top-pp) sampling across various temperature settings.

Without comparing against these inference-time controls, it remains unclear whether Q-ROLE's complex RL framework (learnable soft role embeddings and quality-gated rewards) is necessary or if a simple decoding heuristic could achieve comparable gains with zero training overhead.

Methodological limitation on diversity

The coverage bonus measures divergence strictly relative to the Anchor output. It does not explicitly penalize redundancy among different Explorer paths. While the authors demonstrate empirically, the reward formulation itself lacks a formal pairwise diversity considerations.

Questions for authors

See weaknesses part.

Reviewer_Sv3FOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This study explores how generative recommendations can obtain more useful candidate sets under a fixed candidate budget. The authors first illustrate through industry requests that multiple generation results may share an early prefix, and random exploration can simultaneously increase the maximum predicted reward of the candidates and reduce the average reward. The proposed Q-ROLE learns persistent Anchor and Explorer roles within the same generator: The Explorer only receives coverage rewards related to the fork position of the initial prefix when the base reward reaches the tolerance threshold of the current Anchor; the unselected candidates retain the base reward. The role conditions are used for both training and inference, and ultimately, the exploration reward is not used. The experiments include five host adaptations, candidate quality and mechanism ablation, budget and delay analysis on two public datasets, as well as online industry results over six days. The public evaluation limits the generation within a held-out target and a 999 negative sample pool for each request, so the conclusion is first applicable to the declared evaluation scope.

Critical strengths

The methods and diagnoses are organized around the problem of identifying a clear candidate set. The role division in the shared generator and the quality condition rewards can directly express the goal of "encouraging useful early bifurcations", the single exploration reward is limited to [0,β], and no additional prefix value model is required. The authors also report candidate coverage, base rewards, target achievability, and final hit rate, without considering the single maximum value as the entire effect. The appendix also accurately explains that this score measures the Anchor relative bifurcation, does not guarantee that Explorers are pairwise different, and the gating passes are not equal to positive advantages; these boundaries make the method claim to be more reviewable.

The ablation design is targeted at key alternative explanations, and there is clear evidence of repeated runs for positive results. Apart from using roles alone, gating alone, and no gating rewards, the authors provide random gating with matching actual average bonus, training period shared/independent ID control, and inference intervention that only changes the role input after freezing the same model. The fixed evaluation reference is also separated from the training of the Anchor. The main UGR comparison report of Office shows that the three sub HR@10 range from 7.258% to 7.834%, the paired gain is 0.576 percentage points, and the 95% interval is [0.528, 0.624]; the paired results of five hosts and two datasets are in the same direction. The delay table also includes the final scorer in the total time, supporting the actual advantage without additional value queries under the tested configuration.

Critical weaknesses

The actual calculation of "candidate quality" and the core generation protocol are not yet specific enough, which limits the interpretation of the main evidence. §4 records the base reward as R(x,y), Appendix A defines how to aggregate it, and Appendix B explains how to retain the host optimization and adapt the slate host, but the full text does not provide the specific construction of the actual reward function, scale transformation, and final task evaluator used on Office/Beauty. The approximately 0.7 magnitude Reward Mean, β=0.005, δ=0.01, and quality threshold in Table 2–4 all depend on these definitions, so this exceeds the usual scope of omitting regular hyperparameters. Additionally, "beam expansion and pruning retain role identity" cannot determine the width of each role beam, the cross-role pruning method, and the role allocation when the budget increases, the actual length of SID is also not clear, making it difficult to fully review role coverage and budget matching. The authors have stated that these are incremental comparisons on the adapted task; what needs to be supplemented at present is how the existing experiments are implemented.

The statistical evidence focuses on a main comparison, and the accuracy of the mechanism conclusion has not been fully demonstrated. Appendix C clearly only gives three sub-statistics for Office, UGR, and G=16, and explains that this approach cannot be generalized to other experiments. Part of Table 4's "±" does not clearly correspond to the number of seeds and SD/SE/CI; the HR difference between independent roles and shared roles is 0.165 percentage points, and the difference between complete gating and matching random gating is 0.225 percentage points, but these key comparisons do not provide paired uncertainties. Therefore, the existing materials support a promising point estimate, but it cannot be determined whether the fine-grained mechanism effect is stable against operational fluctuations. The online results are still aggregated point estimates; Appendix G explains the need for unified exposure normalization, but does not explain the actual denominator,diversion unit, and intervals used, so the evidence strength should also be limited.

Can the R, reward scale or normalization used in training/quality diagnosis in public experiments, as well as the precise definition of the final scorer be provided? And use a simple table to explain which supervision and scoring steps are retained or changed in each host after adaptation? Especially, it is necessary to be able to understand Reward Mean in Table 2–4 and gating parameters from the definitions, rather than just explaining the shared evaluator.

Can a brief pseudo-code or equivalent configuration for the role beam search in public experiments be provided, clearly specifying the SID length, the number of learned roles, the budget for each role/global beam, and the mapping of roles and generation opportunities when G=8,16,24,32? These pieces of information will determine how to interpret the budget effect in Figure 4 and the contribution of independent roles to coverage?

Can "±" in Table 4 be defined, and prioritize using existing repeated runs to report the shared/independent roles and the paired differences and uncertainties of the matching random/quality gating pairs in Tables 5 and 12? If there are no repeated runs, a small-scale budget review with the same budget for these two key comparisons can directly affect the judgment. The online part should explain the actual splitting and exposure normalization method; if the existing records cannot support the intervals, it should be clearly limited to descriptive point estimation.

Questions for authors

see Weakness

Reviewer_y17mOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors identify a structural exploration–quality trade-off in candidate generation for generative recommenders trained with reinforcement learning (RL). This trade-off can lead to two failure modes: candidate homogenization, where an emphasis on quality through probability-driven decoding produces candidates with shared SID prefixes, and candidate quality degradation, where an emphasis on diversity through stochastic sampling reduces generation quality.

To better balance these objectives, the authors propose Q-ROLE (Quality-Gated Role Exploration). Specifically, Q-ROLE assigns one generated candidate the Anchor role and the others Explorer roles. It incorporates distinct role embeddings into the conditioning contexts of different Explorers to encourage generation with diverse prefixes. Furthermore, Q-ROLE applies a quality-gated reward bonus to Explorer outputs, rewarding only those with prefixes distinct from the Anchor’s and a limited loss in quality relative to the Anchor.

Experiments demonstrate the effectiveness of Q-ROLE when combined with different recommendation methods. Further analyses validate the quality-gated bonus and distinct role embeddings and indicate favorable scaling and high efficiency.

Critical strengths
  • Novel and Flexible Method. Sampling is an important avenue for improving the performance of generative recommenders trained with RL. Q-ROLE provides a novel approach by assigning different role embeddings to different candidate generation processes. It is also compatible with various recommendation methods, such as GFlowGR [1] and GoalRank [2].
  • Comprehensive Experiments. Q-ROLE improves the performance of different recommendation methods, suggesting its generalizability across these methods. The ablation studies validate the effectiveness of the role assignments and Explorer reward bonuses. Further analyses indicate favorable scaling and high efficiency. Online evaluation demonstrates the practicality of Q-ROLE in industrial settings.

[1] Wang Y, Zhou S, Lu J, et al. Gflowgr: Fine-tuning generative recommendation frameworks with generative flow networks[C]//Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval. 2026: 1925-1936.

[2] Zhang K, Wang X, Liu S, et al. Goalrank: Group-relative optimization for a large ranking model[C]//International Conference on Learning Representations. 2026, 2026: 155825-155845.

Critical weaknesses
  • Limited Evaluation of Diversity across Explorers. Addressing candidate homogenization requires diversity across the candidate group. However, Q-ROLE assigns each Explorer a bonus based on its divergence from the Anchor, without directly accounting for overlap among Explorers. Although Appendix H acknowledges this limitation, the paper could further empirically evaluate an explicit group-level diversity bonus. For example, the kk-th generated candidate could receive a higher bonus when it differs from the first k−1k-1 candidates.

  • Limited Public-Benchmark Coverage and Reliance on a Single Anchor. Despite the industrial evaluation, the public-benchmark experiments cover only two Amazon datasets. Results on additional datasets would further validate the method’s effectiveness. In addition, the training-time quality threshold depends on the reward of a single Anchor, and the sensitivity to this design is not evaluated. It would be useful to analyze the effect of allocating multiple Anchors and using their mean reward to set the reference reward.

Questions for authors

Please refer to the weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.