acceptodds
Under review as a conference paper at ICLR 2027

F-GRPO: Factorized Group Relative Policy Optimization for Unified Candidate Generation and Ranking

Abstract

Traditional retrieval pipelines optimize utility through stages of candidate retrieval and reranking, where ranking operates over a predefined candidate set. Large Language Models (LLMs) broaden this into a generative process: given a candidate pool, an LLM can generate a subset and order it within a single autoregressive pass. However, this flexibility introduces a new optimization challenge: the model must search a combinatorial output space while receiving utility feedback only after the full ranked list is generated. Because this feedback is defined over the completed sequence, it cannot distinguish whether a poor result arises from failing to generate a relevant subset or from failing to rank that subset correctly. This credit assignment gap makes end-to-end optimization unstable and sample-inefficient. Existing systems often address this challenge by separating candidate generation from ranking. However, such decoupling remains misaligned with downstream utility because the ranking stage is fundamentally limited by the candidate set it receives. To bridge the optimization gap between candidate generation and ranking, we propose a unified framework that performs both within a single autoregressive rollout and optimizes them end-to-end via factorized group-relative policy optimization (F-GRPO). Our framework factorizes the policy into candidate generation and ranking while sharing a single LLM backbone across both stages, and jointly trains them with an order-invariant coverage reward and a position-aware utility reward. To address the resulting phase-specific credit assignment problem, we use separate group-relative advantages for generation and ranking within a two-phase sequence-level objective. Across sequential recommendation and multi-hop question answering benchmarks, F-GRPO improves ranking quality over GRPO and decoupled baselines, outperforms supervised alternatives, and exceeds zero-shot rerankers on most metrics, with no architectural changes at inference time.

Primary area:
foundation or frontier models, including LLMs
Keywords:
reinforcement learning, large language models, policy optimization, retrieval and ranking

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_MtrcOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies credit assignment in RL-based fine-tuning of LLM rerankers under two-phase structured rollouts. The authors observe that a single scalar reward in GRPO conflates two distinct objectives: an order-invariant coverage objective (whether the relevant items are selected) and a position-sensitive utility objective (where the selected items are finally placed). They propose F-GRPO, which assigns separate scalar rewards to the two phases via structured output tags and applies per-segment advantages within a single backward pass. Experiments on Qwen backbones (2B/4B) across benchmarks including MovieLens show consistent improvements over GRPO, demonstrating the effectiveness of the proposed F-GRPO.

Critical strengths
  • The paper precisely identifies why standard GRPO is misaligned with two-phase structured rollouts: the coverage objective is order-invariant, whereas the utility objective is position-sensitive, and a single scalar advantage conflates the two. This decomposition is intuitive and directly motivates the method.
  • F-GRPO is simple, elegant, and effective. It yields consistent and competitive gains, demonstrating its effectiveness. It requires only structured output tags, two scalar rewards, and per-segment advantage computation within a single backward pass. F-GRPO can therefore be dropped into existing GRPO pipelines with minimal engineering effort.
  • The experimental setup is thorough and diverse, and the results consistently support the effectiveness of the proposed method, though certain aspects could be further strengthened, as discussed in the Weaknesses section.
Critical weaknesses
  • The experiment part could be better: I found the chosen LLM-based rerankers are a bit weak. I would strongly encourage to compare with strong LLM-based rerankers such as Qwen3-Reranker [1] and ProRank [2], the latter also leverages GRPO to improve document reranking.
  • In Table 1, F-GRPO outperforms GRPO in all settings except MovieLens with the Qwen3.5-2B backbone (whereas the Qwen3-4B counterpart achieves clear absolute gains). This exception raises the concern that the benefit of fine-grained credit assignment may depend on model scale or dataset characteristics. To strengthen the claim, I would suggest to evaluate at least one additional model family (e.g., LLaMA) and, ideally, more scales;

References:

  • [1] https://huggingface.co/collections/Qwen/qwen3-reranker
  • [2] Li, X., Shakir, A., Huang, R., Lipp, J., Clavié, B. and Li, J., 2026, July. Prorank: Prompt warmup via reinforcement learning for small language models reranking. In Findings of the Association for Computational Linguistics: ACL 2026 (pp. 1026-1037).
Questions for authors
  • Why are two different Qwen versions (Qwen3 and Qwen3.5) used? Holding the version fixed would isolate the effect of F-GRPO from version-level differences such as pretraining data and instruction-following ability. If the mixture is due to checkpoint availability, please clarify, and confirm whether the MovieLens exception at 2B is attributable to the version change rather than to F-GRPO itself.
  • The manuscript does not mention code availability. Do the authors plan to release code?
Reviewer_QZasOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes F-GRPO, a reinforcement learning framework that unifies candidate selection and ranking within a single LLM rollout. The method factorizes generation into two phases: constructing a candidate slate and ranking the selected candidates. To address the credit-assignment ambiguity in standard GRPO, F-GRPO introduces separate group-relative advantages based on an order-invariant slate reward (Recall/F1) and a position-aware ranking reward (NDCG). Both phases share the same LLM backbone and are optimized jointly using a Dr. GRPO-based objective. Experiments on two sequential recommendation datasets (LastFM and MovieLens) and two multi-hop QA datasets (MuSiQue and HotpotQA), using Qwen3-4B and Qwen3.5-2B, demonstrate improvements over GRPO and supervised baselines in most settings. Ablation studies further show that phase-specific credit assignment improves performance over a shared-advantage formulation.

Critical strengths
  1. The paper identifies a meaningful credit-assignment issue when candidate selection and ranking are optimized using a single sequence-level reward. The proposed phase-specific advantages offer a simple and intuitive solution.

  2. F-GRPO builds on Dr. GRPO with separate rewards and token-level loss masking, without requiring additional model architectures or separately trained ranking modules. The method is straightforward to implement and potentially applicable to other structured-generation tasks.

  3. The experiments cover four benchmarks spanning recommendation and multi-hop QA, using two LLM backbones. F-GRPO achieves notable improvements over GRPO in several settings, particularly on LastFM and MuSiQue.

  4. The paper includes a matched comparison with Structured Joint GRPO to isolate the effect of factorized credit assignment. Additional analyses of reward formulation, training dynamics, and slate–ranker behavior provide insights into how the two phases contribute to performance.

Critical weaknesses
  1. F-GRPO removes downstream ranking credit from slate tokens despite their influence on ranking quality, the paper lacks a comparison experiment to justify fully separated credit assignment. (assign both slate reward and rank reward to slate token while assigning rank reward to rank token)

  2. Table 3 shows that most LastFM gains come from the two-block output structure (e.g. adding the structured output takes the average from 54.6 to 61.1 while separating advantages only take it from 61.1 to 62.5 for LastFM). It raises the concerns that improvement comes from the output structure rather than factorized credit assignment.

  3. The evaluation setting is simplified: Experiments primarily use 20 candidates containing all gold items, with limited negative-sampling analysis. QA is evaluated only on evidence selection, with no downstream answer accuracy.

  4. F-GRPO underperforms GRPO on 5 of 6 metrics for Qwen3.5-2B MovieLens. Some Decoupled SFT results are even below random ranking, raising concerns about baseline implementation and fairness.

Questions for authors
  1. Why does F-GRPO completely remove downstream ranking credit from slate tokens?
  2. Does the core contribution come from structured output instead of credit assignment design?
Reviewer_EcMnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies subset selection and ranking within a supplied candidate pool, rather than retrieval from an entire corpus. A single LLM first generates a candidate slate in a <SLATE> block and then produces a final ranking of a subset of those candidates in a <RANK> block, all within one autoregressive rollout. F-GRPO computes separate group-mean-centered advantages from a set-level coverage reward and a position-sensitive ranking reward, assigning each advantage to the tokens of its corresponding phase. Both phases share parameters and are trained jointly. Experiments on LastFM, MovieLens, MuSiQue, and HotpotQA using Qwen3-4B and Qwen3.5-2B show improvements over direct-ranking GRPO in most settings.

Critical strengths
  1. Simple, well-motivated formulation with zero architectural overhead: Separating candidate selection and ranking into <SLATE> and <RANK> spans within one rollout directly addresses distinct failure modes (coverage vs. ordering). Both rewards use the same relevance labels without extra annotations or separate models, avoiding the train–test distribution shift of decoupled two-stage pipelines.
  2. Clean isolation of phase-specific credit assignment: The matched ablation against Structured Joint GRPO (Table 3) fixes the two-block output format, initialization, and rewards, changing only whether advantages are shared or separated by span.
Critical weaknesses
  1. Reward–evaluation mismatch and truncated causal credit assignment: Training NDCG in Eq. (12) normalizes by the ideal ordering of only the emitted list τσ\tau_\sigma, whereas evaluation NDCG normalizes by all gold items in the pool G(x)G(x). On multi-gold QA tasks, emitting a single gold passage yields a perfect training rank reward (Rrank=1.0R_{\text{rank}} = 1.0) even if other gold passages from the slate are dropped. This explains why average output lengths on QA collapse to 2.08–2.642.08\text{--}2.64 items (Table 5) and Recall@3/@5 plateau (Table 2). Furthermore, under the joint objective in Eq. (2), the slate action τ\tau causally bounds the downstream ranking return, yet F-GRPO drops downstream credit to slate tokens without comparing against a causal credit baseline (where slate tokens receive Aslate+λArankA_{\text{slate}} + \lambda A_{\text{rank}} and rank tokens receive λArank\lambda A_{\text{rank}}).

  2. Unmatched token/compute overhead and narrow experimental scope: Generating <SLATE> plus <RANK> roughly doubles output tokens compared to Direct-Rank GRPO, yet comparisons omit wall-clock training time, inference latency, and a capacity-matched baseline (e.g., Direct-Rank GRPO outputting 10 items and evaluated at top-kk). Evaluation is also limited to small candidate pools (20 items, or 50 with easy random distractors rather than hard retrieval negatives), small models (2B/4B).

Questions for authors
  1. Training IDCG\text{IDCG} Normalization: Why normalize training IDCG@k\text{IDCG}@k over emitted items τσ\tau_\sigma instead of the full gold set G(x)G(x)? Does normalizing by G(x)G(x) prevent the length collapse to ∼2.1–2.6\sim 2.1\text{--}2.6 items on QA (Table 5) and improve Recall@3/@5?
  2. Causal Credit Baseline: How does F-GRPO compare against an unbiased causal credit baseline where slate tokens receive Aslate+λArankA_{\text{slate}} + \lambda A_{\text{rank}} and rank tokens receive λArank\lambda A_{\text{rank}}?
  3. Compute-Matched Comparison: What are the training GPU-hours and inference latencies of F-GRPO vs. Direct-Rank GRPO, and how does Direct-Rank GRPO perform when trained to output 10 ranked items (matching F-GRPO's slate capacity)?
  4. Fairer Decoupled Baseline: How does the decoupled pipeline perform if its ranker is trained on the selector's predicted slates rather than gold slates?
1 comment

If the authors addressed your concerns or clarified matters, let them know whether your concerns are resolved or questions remain.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.