- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies credit assignment in RL-based fine-tuning of LLM rerankers under two-phase structured rollouts. The authors observe that a single scalar reward in GRPO conflates two distinct objectives: an order-invariant coverage objective (whether the relevant items are selected) and a position-sensitive utility objective (where the selected items are finally placed). They propose F-GRPO, which assigns separate scalar rewards to the two phases via structured output tags and applies per-segment advantages within a single backward pass. Experiments on Qwen backbones (2B/4B) across benchmarks including MovieLens show consistent improvements over GRPO, demonstrating the effectiveness of the proposed F-GRPO.
Critical strengths
- The paper precisely identifies why standard GRPO is misaligned with two-phase structured rollouts: the coverage objective is order-invariant, whereas the utility objective is position-sensitive, and a single scalar advantage conflates the two. This decomposition is intuitive and directly motivates the method.
- F-GRPO is simple, elegant, and effective. It yields consistent and competitive gains, demonstrating its effectiveness. It requires only structured output tags, two scalar rewards, and per-segment advantage computation within a single backward pass. F-GRPO can therefore be dropped into existing GRPO pipelines with minimal engineering effort.
- The experimental setup is thorough and diverse, and the results consistently support the effectiveness of the proposed method, though certain aspects could be further strengthened, as discussed in the Weaknesses section.
Critical weaknesses
- The experiment part could be better: I found the chosen LLM-based rerankers are a bit weak. I would strongly encourage to compare with strong LLM-based rerankers such as Qwen3-Reranker [1] and ProRank [2], the latter also leverages GRPO to improve document reranking.
- In Table 1, F-GRPO outperforms GRPO in all settings except MovieLens with the Qwen3.5-2B backbone (whereas the Qwen3-4B counterpart achieves clear absolute gains). This exception raises the concern that the benefit of fine-grained credit assignment may depend on model scale or dataset characteristics. To strengthen the claim, I would suggest to evaluate at least one additional model family (e.g., LLaMA) and, ideally, more scales;
References:
- [1] https://huggingface.co/collections/Qwen/qwen3-reranker
- [2] Li, X., Shakir, A., Huang, R., Lipp, J., Clavié, B. and Li, J., 2026, July. Prorank: Prompt warmup via reinforcement learning for small language models reranking. In Findings of the Association for Computational Linguistics: ACL 2026 (pp. 1026-1037).
Questions for authors
- Why are two different Qwen versions (Qwen3 and Qwen3.5) used? Holding the version fixed would isolate the effect of F-GRPO from version-level differences such as pretraining data and instruction-following ability. If the mixture is due to checkpoint availability, please clarify, and confirm whether the MovieLens exception at 2B is attributable to the version change rather than to F-GRPO itself.
- The manuscript does not mention code availability. Do the authors plan to release code?
If the authors addressed your concerns or clarified matters, let them know whether your concerns are resolved or questions remain.