- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
The authors investigate the reliability of negative advantages in GRPO-based recommendation model fine-tuning. Recommendations assigned negative advantages may still align with user interests revealed by subsequent interactions.
To address this issue, they propose Risk-Calibrated Negative Suppression (RCNS), which attenuates potentially unreliable negative updates. Their analysis identifies sign conflicts, where recommendations with negative advantages under training feedback receive positive advantages when evaluated against subsequent feedback. They further find that generally better target retrieval ranks are associated with a larger ratio of sign conflicts. RCNS therefore assigns stronger attenuation to these responses without reversing their advantage signs. The attenuation strength is calibrated using validation feedback under an empirical constraint on attenuating non-conflicting negative advantage mass.
Experiments on three datasets show that sign conflicts involve appreciable negative advantage mass and support the association between better target retrieval ranks and higher conflict rates. In the main comparison, RCNS achieves higher mean recommendation performance than GRPO and the other evaluated baselines.
Critical strengths
- Important Problem. The reliability of negative learning signals is an important issue in recommendation. Recommendations assigned negative advantages under training feedback may still align with user interests revealed by subsequent interactions. By introducing the concept of advantage sign conflicts, the authors provide a useful framework for examining this issue.
- Intuitive Method. RCNS offers an intuitive and novel approach that connects the identification of potentially unreliable negatives with their attenuation: it uses target retrieval ranks as a reliability cue and assigns stronger attenuation to negative-advantage responses with better target ranks. By weakening their suppressive effect without reversing advantage signs, it directly mitigates the risk of discouraging recommendations that may still align with user interests.
Critical weaknesses
- Unclear Feedback Requirements. The paper motivates the rank proxy by noting that directly identifying sign conflicts requires an additional feedback set. However, RCNS also requires an additional feedback set to calibrate its attenuation parameter. The method therefore retains the dependence on extra feedback, albeit at the calibration stage. The authors should explain why this requirement is treated as an obstacle to direct conflict identification but is acceptable for parameter calibration, and clarify which feedback-availability constraint the proposed proxy actually resolves.
- Calibration Overhead. Calibrating the attenuation parameter requires additional LLM rollouts. The authors should quantify the computational overhead of this stage, and compare it with the original RL process. It is also advised to clearly specify the source and size of the calibration data, as well as the number of sampled responses.
Questions for authors
Please refer to the weaknesses.
4 comments
Response to Reviewer NoGt, Part 2/2
W2: Calibration overhead, and the source and size of the calibration data.
Source and size of the calibration data. Calibration uses each domain's chronologically split validation feedback, and the rollouts are generated by the pretrained Qwen3-1.7B before RL, with G = 8 recommendations per context, temperature 0.8, and top-p 0.95. The fixed diagnostic pools in Appendix A contain 3,710 / 3,301 / 3,179 users and 29,680 / 26,408 / 25,432 rollouts, of which the 3,090 / 2,576 / 2,119 users with validation feedback are used for calibration; see Table 5.
Calibration overhead. Calibration consists of one-time rollout generation and reward computation, followed by a bisection search on the fixed rollouts. The bisection search takes 80 steps and involves only precomputed advantages and ranks, so its time is negligible. Taking Movies & TV as an example, the measured overhead is as follows:
| Item | Value |
|---|---|
| Contexts / Sampled responses | 3,179 / 25,432 |
| Wall-clock time on a single A800 | 11.84 min |
| One-time cost | 0.197 GPU·h |
| Relative to a single GRPO training run | 2.17% |
| Amortized over 5 seeds | 0.43% / run |
Calibration is performed once for each domain and reward configuration, and the same p⋆ is shared by all training seeds under that configuration, which yields the amortized cost of 0.43% per run.
Training overhead. RCNS computes the complete-catalog rank from the same similarity scores already produced for the reward, and training only needs to sort negative-advantage samples within each batch. Measured training costs:
| Dataset | GRPO | RCNS | Relative change |
|---|---|---|---|
| C&V | 6.59 GPU·h | 6.64 GPU·h | +0.7% |
| VG | 6.72 GPU·h | 6.62 GPU·h | −1.4% |
| M&TV | 9.11 GPU·h | 8.94 GPU·h | −1.9% |
The differences are within ±2%, which is normal run-to-run variation; that is, RCNS introduces no observable training overhead.
Cost and benefit. Overall, the only additional cost of RCNS is a one-time offline calibration of about 2% of a single training run. To evaluate the benefit obtained at this cost, we conducted 20 additional independent training runs per domain, pairing GRPO and RCNS under the same seeds. Performance differences follow the metric definitions and the ×10³ display units of Table 18 in Appendix G, with positive values favoring RCNS. The 95% confidence intervals are paired t-intervals with n = 20, 19 degrees of freedom, and t₀.₉₇₅ = 2.093:
| Dataset | Metric | Mean Δ | 95% CI | Relative Gain |
|---|---|---|---|---|
| C&V | HR@5 | +0.213 | [+0.0002, +0.426] | +8.94% |
| C&V | NDCG@5 | +0.208 | [+0.074, +0.343] | +13.24% |
| C&V | HR@10 | +0.185 | [−0.139, +0.509] | +4.64% |
| C&V | NDCG@10 | +0.201 | [+0.043, +0.359] | +9.61% |
| M&TV | HR@5 | +1.956 | [+1.051, +2.861] | +24.81% |
| M&TV | NDCG@5 | +1.337 | [+0.623, +2.052] | +25.24% |
| M&TV | HR@10 | +2.718 | [+1.465, +3.971] | +22.73% |
| M&TV | NDCG@10 | +1.579 | [+0.768, +2.390] | +23.95% |
| VG | HR@5 | +0.477 | [+0.089, +0.866] | +4.12% |
| VG | NDCG@5 | +0.331 | [+0.009, +0.654] | +4.36% |
| VG | HR@10 | +0.628 | [+0.020, +1.237] | +3.51% |
| VG | NDCG@10 | +0.375 | [+0.007, +0.743] | +3.90% |
In the 20 additional paired runs, RCNS achieves a positive mean improvement on all 12 metrics, with relative gains of 3.51%–25.24%, and the 95% paired t-intervals lie above zero for 11 of them. A one-time offline calibration of about 2% of a training run thus yields consistent improvements in recommendation performance. The revision will include the complete overhead analysis and a description of the calibration data.
We thank the reviewer again for the constructive comments.
Thank the authors for their detailed responses. The authors further clarify the problems of directly using conflict labels, which makes the proposed rank proxy more reasonable. Besides, the extra costs of calibration and RCNS training are also acceptable. Given this, I improve the soundness score and the rating to 3.
We sincerely thank you for your insightful comments and are glad that our responses have addressed your concerns. We truly appreciate your time, and we wish you all the best!
Response to Reviewer NoGt, Part 1/2
We thank the reviewer for recognizing the importance of the problem we study and the intuitiveness and novelty of RCNS. Below we address the reviewer's concerns about feedback requirements and calibration overhead.
W1: The feedback requirements are unclear, since calibration also relies on additional feedback.
RCNS converts the per-sample, training-time requirement for subsequent feedback into a one-time, domain-level parameter calibration before training: Retrieval Rank determines which negative updates are attenuated first, and validation feedback only determines the attenuation strength. Retrieval Rank comes directly from reward computation, is comparable across groups within the entire batch, and requires no additional feedback during training. Sign Conflict, in turn, is used to diagnose the problem, to validate the effectiveness of Rank, and to select p⋆ on validation feedback before training; it is not an input to RCNS during training. Directly identifying conflicts and calibrating RCNS differ fundamentally in their feedback requirements in three respects:
With the Retrieval Rank proxy, the need for subsequent feedback on every sample at every training step becomes a single scalar determined once before training from the standard validation split. This works because the calibration remains valid throughout training. Table 15 in Appendix D.3 shows that p₀, calibrated on pretrained rollouts and held fixed throughout training, keeps the test FCR within one percentage point of the 10% validation FCR budget in all three domains after both training passes, with final-policy RGC of 13.57%–19.72%, without checkpoint-specific recalibration. Furthermore, Appendix D.1 shows that the standard deviation of p⋆ across five independent rollout pools is only 0.36–0.56, indicating that the calibration is stable. In the revision, we will clearly distinguish these two types of feedback requirements in the introduction and the method section.
References
[1] Mohamed Sana, Nicola Piovesan, Antonio De Domenico, Fadhel Ayed, and Haozhe Zhang. HPO: Hysteretic Policy Optimization for Stable and Efficient Training under Sparse-Reward Regime. arXiv:2605.30201, 2026.