acceptodds
Under review as a conference paper at ICLR 2027

When Negative Advantages Are Unreliable: Risk-Calibrated Negative Suppression for Generative Recommendation

Abstract

In generative recommendation, reinforcement learning (RL) optimizes recommendation policies using rewards derived from user feedback. Methods such as GRPO compute advantages from the relative rewards of multiple recommendations generated for the same context and suppress those with negative advantages. However, the user feedback available during training is incomplete: recommendations unsupported by observed feedback may still match user interests, so suppressing them risks discouraging otherwise plausible recommendations. To investigate this problem, we measure the fraction of negative-advantage recommendations whose advantages become positive when evaluated against subsequent feedback. Across three recommendation domains, 5.2%–13.1% of recommendations with negative observed advantages receive positive advantages under validation feedback, demonstrating the recurrence of these sign conflicts. This motivates selectively attenuating feedback-sensitive negative updates while retaining useful negative learning signals. The challenge is to determine which updates to attenuate using information available during training. We find that sign conflicts are more likely among negative-advantage recommendations with better observed retrieval ranks, providing an observable cue available during training. Building on this insight, we propose Risk-Calibrated Negative Suppression (RCNS), which reuses this ordering to allocate continuous attenuation without reversing advantage signs. Using held-out validation feedback, we calibrate a shared domain-level attenuation parameter to maximize coverage of sign-conflicting advantage mass subject to an empirical risk budget on attenuating non-conflicting negative mass. The calibrated parameter remains fixed during policy training and requires no subsequent feedback for newly generated recommendations. Offline experiments show that RCNS achieves greater held-out advantage-weighted coverage of sign-conflicting updates than uniform attenuation under the same validation risk budget. In the main downstream comparison, RCNS improves mean HR@5 across five independent seeds over GRPO by 5.0%–19.7% across three domains.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Generative Recommendation, Reinforcement Learning, Negative-Advantage Reliability, Risk-Calibrated Policy Optimization

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_NoGtOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

The authors investigate the reliability of negative advantages in GRPO-based recommendation model fine-tuning. Recommendations assigned negative advantages may still align with user interests revealed by subsequent interactions.

To address this issue, they propose Risk-Calibrated Negative Suppression (RCNS), which attenuates potentially unreliable negative updates. Their analysis identifies sign conflicts, where recommendations with negative advantages under training feedback receive positive advantages when evaluated against subsequent feedback. They further find that generally better target retrieval ranks are associated with a larger ratio of sign conflicts. RCNS therefore assigns stronger attenuation to these responses without reversing their advantage signs. The attenuation strength is calibrated using validation feedback under an empirical constraint on attenuating non-conflicting negative advantage mass.

Experiments on three datasets show that sign conflicts involve appreciable negative advantage mass and support the association between better target retrieval ranks and higher conflict rates. In the main comparison, RCNS achieves higher mean recommendation performance than GRPO and the other evaluated baselines.

Critical strengths
  • Important Problem. The reliability of negative learning signals is an important issue in recommendation. Recommendations assigned negative advantages under training feedback may still align with user interests revealed by subsequent interactions. By introducing the concept of advantage sign conflicts, the authors provide a useful framework for examining this issue.
  • Intuitive Method. RCNS offers an intuitive and novel approach that connects the identification of potentially unreliable negatives with their attenuation: it uses target retrieval ranks as a reliability cue and assigns stronger attenuation to negative-advantage responses with better target ranks. By weakening their suppressive effect without reversing advantage signs, it directly mitigates the risk of discouraging recommendations that may still align with user interests.
Critical weaknesses
  • Unclear Feedback Requirements. The paper motivates the rank proxy by noting that directly identifying sign conflicts requires an additional feedback set. However, RCNS also requires an additional feedback set to calibrate its attenuation parameter. The method therefore retains the dependence on extra feedback, albeit at the calibration stage. The authors should explain why this requirement is treated as an obstacle to direct conflict identification but is acceptable for parameter calibration, and clarify which feedback-availability constraint the proposed proxy actually resolves.
  • Calibration Overhead. Calibrating the attenuation parameter requires additional LLM rollouts. The authors should quantify the computational overhead of this stage, and compare it with the original RL process. It is also advised to clearly specify the source and size of the calibration data, as well as the number of sampled responses.
Questions for authors

Please refer to the weaknesses.

4 comments
AuthorsOpenReview ↗

Response to Reviewer NoGt, Part 1/2

We thank the reviewer for recognizing the importance of the problem we study and the intuitiveness and novelty of RCNS. Below we address the reviewer's concerns about feedback requirements and calibration overhead.

W1: The feedback requirements are unclear, since calibration also relies on additional feedback.

RCNS converts the per-sample, training-time requirement for subsequent feedback into a one-time, domain-level parameter calibration before training: Retrieval Rank determines which negative updates are attenuated first, and validation feedback only determines the attenuation strength. Retrieval Rank comes directly from reward computation, is comparable across groups within the entire batch, and requires no additional feedback during training. Sign Conflict, in turn, is used to diagnose the problem, to validate the effectiveness of Rank, and to select p⋆ on validation feedback before training; it is not an input to RCNS during training. Directly identifying conflicts and calibrating RCNS differ fundamentally in their feedback requirements in three respects:

  • Granularity. Directly identifying sign conflicts requires subsequent feedback for every newly generated recommendation during training and recomputing its advantage, i.e., per-sample conflict labels. Calibrating RCNS only requires determining a single domain-level scalar p⋆ from validation feedback. Like the learning rate or the attenuation coefficient c in HPO [1], this is a standard hyperparameter selection on the validation set.
  • Coverage. Subsequent feedback exists only for part of the training users: the fixed diagnostic pools include every policy-training user with feedback in a later split, and among the 8,012 / 7,763 / 8,993 policy-training users, only 3,090 / 2,576 / 2,119 have validation feedback. Per-sample conflict labels are therefore unavailable for most training samples, whereas Retrieval Rank is available for every generated recommendation with a valid title, and the calibrated scalar applies to all of them.
  • How the data are used. If validation feedback were used directly to label training rollouts, validation information would enter every gradient update. Calibration uses held-out validation feedback to estimate a single domain-level scalar, and only this scalar, rather than per-sample conflict labels, enters policy training; test feedback is never involved.

With the Retrieval Rank proxy, the need for subsequent feedback on every sample at every training step becomes a single scalar determined once before training from the standard validation split. This works because the calibration remains valid throughout training. Table 15 in Appendix D.3 shows that p₀, calibrated on pretrained rollouts and held fixed throughout training, keeps the test FCR within one percentage point of the 10% validation FCR budget in all three domains after both training passes, with final-policy RGC of 13.57%–19.72%, without checkpoint-specific recalibration. Furthermore, Appendix D.1 shows that the standard deviation of p⋆ across five independent rollout pools is only 0.36–0.56, indicating that the calibration is stable. In the revision, we will clearly distinguish these two types of feedback requirements in the introduction and the method section.

References

[1] Mohamed Sana, Nicola Piovesan, Antonio De Domenico, Fadhel Ayed, and Haozhe Zhang. HPO: Hysteretic Policy Optimization for Stable and Efficient Training under Sparse-Reward Regime. arXiv:2605.30201, 2026.

AuthorsOpenReview ↗

Response to Reviewer NoGt, Part 2/2

W2: Calibration overhead, and the source and size of the calibration data.

Source and size of the calibration data. Calibration uses each domain's chronologically split validation feedback, and the rollouts are generated by the pretrained Qwen3-1.7B before RL, with G = 8 recommendations per context, temperature 0.8, and top-p 0.95. The fixed diagnostic pools in Appendix A contain 3,710 / 3,301 / 3,179 users and 29,680 / 26,408 / 25,432 rollouts, of which the 3,090 / 2,576 / 2,119 users with validation feedback are used for calibration; see Table 5.

Calibration overhead. Calibration consists of one-time rollout generation and reward computation, followed by a bisection search on the fixed rollouts. The bisection search takes 80 steps and involves only precomputed advantages and ranks, so its time is negligible. Taking Movies & TV as an example, the measured overhead is as follows:

ItemValue
Contexts / Sampled responses3,179 / 25,432
Wall-clock time on a single A80011.84 min
One-time cost0.197 GPU·h
Relative to a single GRPO training run2.17%
Amortized over 5 seeds0.43% / run

Calibration is performed once for each domain and reward configuration, and the same p⋆ is shared by all training seeds under that configuration, which yields the amortized cost of 0.43% per run.

Training overhead. RCNS computes the complete-catalog rank from the same similarity scores already produced for the reward, and training only needs to sort negative-advantage samples within each batch. Measured training costs:

DatasetGRPORCNSRelative change
C&V6.59 GPU·h6.64 GPU·h+0.7%
VG6.72 GPU·h6.62 GPU·h−1.4%
M&TV9.11 GPU·h8.94 GPU·h−1.9%

The differences are within ±2%, which is normal run-to-run variation; that is, RCNS introduces no observable training overhead.

Cost and benefit. Overall, the only additional cost of RCNS is a one-time offline calibration of about 2% of a single training run. To evaluate the benefit obtained at this cost, we conducted 20 additional independent training runs per domain, pairing GRPO and RCNS under the same seeds. Performance differences follow the metric definitions and the ×10³ display units of Table 18 in Appendix G, with positive values favoring RCNS. The 95% confidence intervals are paired t-intervals with n = 20, 19 degrees of freedom, and t₀.₉₇₅ = 2.093:

DatasetMetricMean Δ95% CIRelative Gain
C&VHR@5+0.213[+0.0002, +0.426]+8.94%
C&VNDCG@5+0.208[+0.074, +0.343]+13.24%
C&VHR@10+0.185[−0.139, +0.509]+4.64%
C&VNDCG@10+0.201[+0.043, +0.359]+9.61%
M&TVHR@5+1.956[+1.051, +2.861]+24.81%
M&TVNDCG@5+1.337[+0.623, +2.052]+25.24%
M&TVHR@10+2.718[+1.465, +3.971]+22.73%
M&TVNDCG@10+1.579[+0.768, +2.390]+23.95%
VGHR@5+0.477[+0.089, +0.866]+4.12%
VGNDCG@5+0.331[+0.009, +0.654]+4.36%
VGHR@10+0.628[+0.020, +1.237]+3.51%
VGNDCG@10+0.375[+0.007, +0.743]+3.90%

In the 20 additional paired runs, RCNS achieves a positive mean improvement on all 12 metrics, with relative gains of 3.51%–25.24%, and the 95% paired t-intervals lie above zero for 11 of them. A one-time offline calibration of about 2% of a training run thus yields consistent improvements in recommendation performance. The revision will include the complete overhead analysis and a description of the calibration data.

We thank the reviewer again for the constructive comments.

Thank the authors for their detailed responses. The authors further clarify the problems of directly using conflict labels, which makes the proposed rank proxy more reasonable. Besides, the extra costs of calibration and RCNS training are also acceptable. Given this, I improve the soundness score and the rating to 3.

We sincerely thank you for your insightful comments and are glad that our responses have addressed your concerns. We truly appreciate your time, and we wish you all the best!

Reviewer_YyquOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the negative-advantage problem in GRPO for generative recommendation. Under incomplete training feedback, the authors find that some recommendations assigned negative advantages based on observed feedback receive positive advantages when evaluated using subsequent interactions, suggesting that the original negative training signals may not reliably determine which recommendations should be suppressed. Furthermore, after matching current negative-advantage magnitudes, better-ranked recommendations still exhibit a higher proportion of negative-to-positive advantage flips under subsequent feedback. To address this phenomenon, the authors propose RCNS, which uses retrieval ranks to assign attenuation priorities to negative-advantage recommendations and calibrates the attenuation parameter on the validation set. While limiting the attenuation of negative updates without sign conflicts, RCNS selectively weakens updates more likely to exhibit sign conflicts, thereby reducing the risk of excessively suppressing potentially useful recommendations. Experiments across three recommendation domains show that, under the same validation risk budget, RCNS allocates attenuation more effectively to sign-conflicting updates and outperforms GRPO in mean downstream recommendation metrics.

Critical strengths
  1. The paper proposes a rank-guided selective attenuation mechanism supported by empirical analysis. By holding recommendations and their groups fixed, the authors find that better-ranked recommendations exhibit a higher proportion of sign conflicts even after matching negative-advantage magnitudes. Based on this finding, RCNS constructs batch-relative attenuation priorities and uses a continuous attenuation function that does not reverse advantage signs to selectively weaken potentially unreliable negative updates, providing a targeted improvement to GRPO optimization under incomplete feedback.
  2. RCNS provides an explicit risk-calibration criterion for attenuation strength and is easy to integrate. Using the validation set, the method maximizes coverage of sign-conflicting advantage mass while constraining the fraction of non-conflicting negative-advantage mass attenuated. Only one shared parameter needs to be calibrated per domain, and it remains fixed during subsequent training. No subsequent feedback is required for newly generated recommendations, offering implementation simplicity and practical applicability.
Critical weaknesses
  1. The effects of the feedback definition on calibration and downstream gains remain unclear. Appendix A.1 confirms that sign conflicts and the rank association persist under single-target feedback, but it does not report the parameters or downstream results obtained by recalibrating RCNS using single-target feedback. Since the feedback set directly determines conflict labels and calibration constraints, the persistence of the diagnostic phenomenon across feedback definitions does not automatically establish that the selected parameters and final gains are equally robust.
  2. The downstream gains have substantial statistical uncertainty. All RCNS–GRPO paired differences in Table 18 have 95% bootstrap intervals that include zero. Although all mean differences are positive, the current results do not sufficiently support a conclusion of stable improvement. Furthermore, the paper does not report paired differences or their uncertainty for RCNS versus the random and linear allocation strategies. Therefore, whether its advantage over simpler attenuation strategies is reliable requires further verification.
Questions for authors
  1. In Table 1, SASRec performs substantially worse than other sequential recommendation baselines in some domains, while Caser also shows considerable performance differences across domains. The results are unusually low, approximately at the level of random performance. How were these experimental results obtained? There seem to be some issues with the metrics.
  2. Appendix A.1 confirms that sign conflicts and the rank association persist under single-target feedback. Have the authors further attempted to recalibrate RCNS using single-target feedback? How do the resulting parameters and downstream recommendation performance differ from those obtained using the full feedback window?
5 comments
AuthorsOpenReview ↗

Response to Reviewer Yyqu, Part 1/3

We thank the reviewer for recognizing the rank-guided selective attenuation mechanism of RCNS, its explicit risk-calibration criterion, and its ease of integration. As the reviewer noted, RCNS calibrates only one parameter per domain and requires no subsequent feedback for newly generated recommendations during training. We respond to each point below.

W1 / Q2: The effect of the feedback definition on calibration and downstream gains, and recalibration under single-target feedback.

User feedback in recommender systems is inherently incomplete: the absence of an observed interaction may stem from insufficient exposure, delayed feedback, or incomplete observation, and does not necessarily indicate a lack of interest [1–3]; a successful conversion may also require multiple exposures [4]. Our work identifies that in reinforcement learning for generative recommendation, such weak negative feedback is converted directly into negative policy updates through group-relative advantages. Sign Conflict is used to characterize the reliability of these negative updates and, on that basis, to identify an attenuation-priority signal observable during training. Because feedback is inherently incomplete, no feedback window can fully reflect user preference. We adopt the full window to exploit richer subsequent behavior, and RCNS does not use any Sign Conflict labels during training.

We agree that the persistence of the diagnostic phenomenon across feedback definitions does not automatically imply that the calibrated parameter and final gains are equally robust. We therefore recalibrated RCNS using single-target feedback, i.e., k = 1, with all other settings unchanged and a validation FCR budget of 10%:

DatasetFull-window p⋆k = 1 p⋆Δp⋆Test RGC / FCR with k = 1 calibrationTest RGC / FCR of full-window p⋆ under k = 1
C&V9.319.27−0.0421.50% / 10.60%21.43% / 10.56%
VG8.268.33+0.0714.12% / 10.93%14.23% / 11.01%
M&TV8.899.08+0.1922.79% / 9.80%23.14% / 9.98%

The calibrated p⋆ differs by at most 0.19 between the two feedback definitions. When the full-window p⋆ is applied directly under single-target feedback, its RGC and FCR are nearly identical to those of the recalibrated parameter. Thus, although different feedback windows change the specific conflict labels, the two feedback definitions yield highly similar calibrated parameters and attenuation behavior in all three tested domains.

Similar parameters do not directly imply similar downstream performance, so we further trained RCNS with the single-target-calibrated p⋆ on Movies & TV over five seeds, in units of ×10³:

MethodHR@5NDCG@5HR@10NDCG@10
GRPO8.30 ± 0.695.53 ± 0.6411.76 ± 1.596.64 ± 0.91
RCNS, full-window calibration9.93 ± 2.046.61 ± 1.4714.49 ± 2.588.06 ± 1.62
RCNS, k = 1 calibration9.07 ± 0.356.11 ± 0.2014.30 ± 1.047.75 ± 0.42

RCNS calibrated with single-target feedback still outperforms GRPO on all four metrics, with relative gains of 9.28%–21.60% and noticeably smaller variance. These results show that the gains of RCNS are robust to the feedback definition.

References

[1] Dawen Liang, Laurent Charlin, James McInerney, and David M. Blei. Modeling User Exposure in Recommendation. WWW, 2016.

[2] Yuta Saito. Unbiased Pairwise Learning from Biased Implicit Feedback. ICTIR, 2020.

[3] Jia-Qi Yang, Xiang Li, Shuguang Han, Tao Zhuang, De-Chuan Zhan, Xiaoyi Zeng, and Bin Tong. Capturing Delayed Feedback in Conversion Rate Prediction via Elapsed-Time Sampling. AAAI, 2021.

[4] Xuhui Shao and Lexin Li. Data-driven Multi-touch Attribution Models. KDD, 2011.

AuthorsOpenReview ↗

Response to Reviewer Yyqu, Part 2/3

W2: Statistical uncertainty of downstream gains, and comparison with Random and Linear.

RCNS versus GRPO. To further assess the statistical reliability of the downstream improvements, we conducted 20 additional independent training runs per domain, pairing GRPO and RCNS under the same seeds. Performance differences follow the metric definitions and the ×10³ display units of Table 18 in Appendix G, with positive values favoring RCNS. The 95% confidence intervals are paired t-intervals with n = 20, 19 degrees of freedom, and t₀.₉₇₅ = 2.093:

DatasetMetricMean Δ95% CIRelative Gain
C&VHR@5+0.213[+0.0002, +0.426]+8.94%
C&VNDCG@5+0.208[+0.074, +0.343]+13.24%
C&VHR@10+0.185[−0.139, +0.509]+4.64%
C&VNDCG@10+0.201[+0.043, +0.359]+9.61%
M&TVHR@5+1.956[+1.051, +2.861]+24.81%
M&TVNDCG@5+1.337[+0.623, +2.052]+25.24%
M&TVHR@10+2.718[+1.465, +3.971]+22.73%
M&TVNDCG@10+1.579[+0.768, +2.390]+23.95%
VGHR@5+0.477[+0.089, +0.866]+4.12%
VGNDCG@5+0.331[+0.009, +0.654]+4.36%
VGHR@10+0.628[+0.020, +1.237]+3.51%
VGNDCG@10+0.375[+0.007, +0.743]+3.90%

In the 20 additional paired runs, RCNS achieves a positive mean improvement on all 12 metrics, and the 95% paired t-intervals lie above zero for 11 of them; only the interval for HR@10 on C&V crosses zero.

RCNS versus Random and Linear. Random and Linear are control variants within the RCNS framework: they share the same problem formulation, the same risk calibration, and the same sign-preserving attenuation mechanism as RCNS, and differ only in how attenuation is allocated. Random replaces the rank priority with a random priority, while Linear keeps the rank priority but without nonlinear concentration. Table 2 in the main paper shows that, at a 10% budget, Random and Linear both outperform GRPO on Movies & TV but are comparable to or slightly below GRPO on C&V, while undifferentiated Uniform attenuation underperforms GRPO in both domains. This indicates that attenuation by itself does not necessarily bring gains; the key is which samples the limited attenuation should be allocated to, which is exactly the question RCNS addresses.

The most direct measure of allocation efficiency is the offline RGC: with fixed rollouts and the same FCR budget, it compares how much sign-conflicting negative-advantage mass each method covers, thereby removing the randomness of policy training. At a 10% validation FCR budget, results over five independent rollout pools show RGC of 20.07% / 16.50% / 22.34% for RCNS, 12.57% / 12.62% / 13.42% for Linear, and 9.6% / 9.7% / 11.3% for Random, with all methods having FCR around 10%; see Figure 6 and Appendix D. In other words, at nearly the same cost, RCNS covers about 1.3–1.7 times as many conflicting updates as Linear and about 1.7–2.1 times as many as Random.

The downstream comparisons also favor RCNS. We conducted 20 paired training runs on Movies & TV, comparing RCNS with Uniform, Random, and Linear under the same seeds. Performance differences are reported in units of ×10³, and the 95% confidence intervals are paired t-intervals with n = 20 and 19 degrees of freedom:

ComparisonMetricMean Δ95% CIPositive
RCNS − UniformHR@5+3.403[+2.894, +3.911]20/20
NDCG@5+2.321[+2.028, +2.614]20/20
HR@10+4.881[+4.001, +5.762]20/20
NDCG@10+2.788[+2.388, +3.188]20/20
RCNS − RandomHR@5+1.462[+0.840, +2.085]17/20
NDCG@5+0.879[+0.303, +1.455]14/20
HR@10+1.959[+1.129, +2.790]17/20
NDCG@10+1.035[+0.393, +1.677]15/20
RCNS − LinearHR@5+1.644[+0.528, +2.760]13/20
NDCG@5+0.992[+0.159, +1.824]12/20
HR@10+2.256[+0.412, +4.100]11/20
NDCG@10+1.186[+0.139, +2.233]12/20

The mean differences between RCNS and all three attenuation variants are positive, and all 95% paired t-intervals lie above zero.

AuthorsOpenReview ↗

Response to Reviewer Yyqu, Part 3/3

Q1: The results of SASRec and Caser are unusually low.

We thank the reviewer for the attention to the baseline results. All methods are evaluated against the same full item catalog, which contains a large number of candidate items, so absolute values are low for all methods.

In the original experiments, to be consistent with the prediction setting of the RL policy, we used a sequence-to-one training objective for all ID-based sequential baselines, i.e., each sequence predicts the target item only from its last valid position. This setting fits GRU4Rec and Caser naturally, but the standard implementation of SASRec relies on position-wise supervision, predicting the next item at every valid position of the sequence. Under the sequence-to-one setting, SASRec receives only about 1/12 to 1/20 of the effective loss terms per epoch compared with the position-wise setting.

We therefore ran additional experiments with position-wise supervision for all three baselines, again over five seeds. GRU4Rec and Caser degrade substantially on all metrics under this setting; for example, HR@5 drops from 2.70 / 6.12 / 4.12 to 0.82 / 4.19 / 1.11 for GRU4Rec and from 0.90 / 5.85 / 1.66 to 0.18 / 0.24 / 0.17 for Caser, so of the two objectives compared, the original sequence-to-one setting suits these two models better. The results of SASRec are as follows, in units of ×10³:

DatasetObjectiveHR@5NDCG@5HR@10NDCG@10
C&VSequence-to-one0.19 ± 0.060.15 ± 0.040.42 ± 0.070.22 ± 0.01
C&VPosition-wise0.65 ± 0.240.38 ± 0.131.36 ± 0.370.60 ± 0.15
VGSequence-to-one0.09 ± 0.040.05 ± 0.012.88 ± 0.110.97 ± 0.03
VGPosition-wise3.86 ± 0.292.29 ± 0.207.11 ± 0.353.34 ± 0.13
M&TVSequence-to-one0.70 ± 0.390.54 ± 0.360.83 ± 0.340.58 ± 0.35
M&TVPosition-wise0.62 ± 0.170.40 ± 0.091.35 ± 0.430.63 ± 0.16

With position-wise supervision, SASRec improves substantially on CDs & Vinyl and Video Games, with HR@5 on Video Games increasing from 0.09 to 3.86, while the results on Movies & TV are mixed. RCNS still outperforms all sequential recommendation baselines on every metric in every domain, consistent with Table 1. The cross-domain variation of Caser follows the same trend as GRU4Rec: both perform best on Video Games, the domain with the smallest item catalog. In the revision, we will update the SASRec results, and we commit to releasing our full code upon acceptance.

We thank the reviewer again for the constructive comments.

Reviewer_Sou8OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the reliability of negative advantages in reinforcement-learning-based generative recommendation under incomplete user feedback. Specifically, the paper proposes Risk-Calibrated Negative Suppression (RCNS), which uses retrieval rank to assign recommendation-specific attenuation to negative advantages while preserving their signs. Experiments on three Amazon Reviews domains show the effectiveness of proposed RCNS.

Critical strengths

1.The paper identifies a meaningful failure mode in advantage-based optimization for generative recommendation. For instance, a negative group-relative advantage does not necessarily constitute reliable evidence that a recommendation should be suppressed when user feedback is incomplete. This problem is well motivated and practically relevant.

2.The empirical investigation is structured around the proposed hypothesis, which make the empirical finding considerably more convincing than a simple aggregate correlation.

3.This methods observes that better-ranked negative-advantage recommendations exhibit more sign conflicts. Hence, it provides a training signal that is available without future feedback.

4.The authors provide dataset preprocessing, chronological splits, reward construction, rollout settings, optimization details, etc. Thus, the paper is reasonably reproducible for an RL-based recommendation study.

5.The authors first show that negative advantages can be unstable, then identify retrieval rank as a useful signal, and finally design RCNS around this finding. This progression makes the method feel well motivated rather than heuristic.

Critical weaknesses

1.It is better to distinguish relative advantage instability from genuine improvement in the recommendation itself, and strengthen the raw-reward analysis accordingly. The main reason is that a negative-to-positive sign change does not necessarily mean that the original suppression was wrong.

2.RCNS relies on several empirically motivated design choices with limited theoretical justification. Especially, (1) why retrieval rank should indicate unreliable negative updates; (2) why optimizing RGC under an FCR constraint should improve policy optimization, remain unclear. The authors are encouraged to provide stronger theoretical or mechanistic analysis for these connections.

3.Please add more direct baselines based on confidence-aware or uncertainty-aware advantage weighting to better isolate the contribution of the proposed design.

4.The downstream gains show considerable uncertainty. Although RCNS achieves higher mean performance than GRPO, all paired 95% confidence intervals in “Appendix G” include zero. Hencem additional training seeds would help determine whether these improvements are robust to training randomness.

5.All downstream experiments are based on Amazon domains with retrieval-based rewards. Adding more datasets or setting with a different feedback or reward construction would make the generality of RCNS more convincing.

6.RCNS depends on retrieval ranking for generated recommendations. However, the additional cost is not clearly quantified. It is better to report the extra training-time or retrieval overhead relative to standard GRPO.

7.The current definition uses (|A_i|) as a proxy for update magnitude. Please justify this approximation more clearly or provide a small gradient-level analysis to verify that the same conclusion holds with actual policy-gradient magnitudes.

8.The current Uniform, Random, Linear, and HPO comparisons do not fully disentangle whether RCNS benefits from modeling negative-update reliability or simply from reweighting negative samples. Please consider baselines that weight negative advantages using (|A_i|), reward margin, or model confidence under the same information budget as RCNS.

9.It may be interesting to study whether negative-update reliability can be estimated directly from multiple uncertainty signals rather than relying on retrieval rank alone.

Questions for authors

see weaknesses

7 comments
AuthorsOpenReview ↗

Response to Reviewer Sou8, Part 1/5

We thank the reviewer for recognizing our research motivation, problem diagnosis, method design, and reproducibility. We respond to each point below.

W1: A sign flip does not necessarily mean that the original suppression was wrong.

We thank the reviewer for this important distinction. Sign Conflict characterizes how sensitive a negative update is to changes in the feedback set, and does not by itself mean that the original update was wrong. This distinction is the starting point of our study. In recommender systems, the absence of an observed interaction may stem from insufficient exposure, delayed feedback, or incomplete observation, and does not necessarily indicate a lack of interest. In industrial settings, a successful conversion may also require multiple exposures [1]. Extensive prior work has addressed this issue from the perspectives of exposure modeling and delayed feedback [2–4]. In other words, the feedback observed at training time is only an incomplete snapshot of user preference.

Our work identifies a new way in which this problem affects policy optimization in reinforcement learning for generative recommendation. The model first generates a group of candidates and then computes group-relative advantages from the currently observed feedback. Samples that carry only weak negative feedback are thus converted directly into explicit negative policy updates, which keep lowering the probability that the model generates these recommendations. When positive feedback is already extremely sparse [5–7], such suppression based on incomplete evidence can make potentially valuable recommendations harder to re-explore. Our focus is therefore the reliability of negative updates under incomplete feedback: Sign Conflict measures whether the same negative update still holds under subsequent feedback.

To further distinguish changes in relative advantage from improvements in the recommendation itself, Appendix A.1 already reports an analysis based on the raw retrieval reward. Under single-target test feedback, after one-to-one matching on the magnitude of the observed negative advantage, better-ranked negative-advantage samples have a higher probability of raw retrieval reward improvement in all three domains. As shown in Table 7, after pooling the three domains, the raw-reward improvement rates of high- and low-priority samples are 11.34% and 3.01%, respectively, a difference of 8.33 percentage points with a 95% CI of [4.84, 12.57]. This shows that Retrieval Rank is also associated with subsequent improvements in the raw reward, not only with sign changes of the within-group relative advantage.

This analysis also motivates the continuous, sign-preserving attenuation in RCNS. Since not every potentially conflicting sample should be positively reinforced, RCNS does not flip negative advantages; it selectively weakens negative updates that are more likely to be unreliable according to Retrieval Rank, while the FCR constraint controls the attenuation of non-conflicting negative signals. Accordingly, Sign Conflict diagnoses the optimization risk caused by incomplete feedback, the raw-reward analysis provides more direct evidence on recommendation quality, and the final downstream metrics such as HR and NDCG test whether this mechanism translates into actual recommendation gains.

References

[1] Xuhui Shao and Lexin Li. Data-driven Multi-touch Attribution Models. KDD, 2011.

[2] Dawen Liang, Laurent Charlin, James McInerney, and David M. Blei. Modeling User Exposure in Recommendation. WWW, 2016.

[3] Yuta Saito. Unbiased Pairwise Learning from Biased Implicit Feedback. ICTIR, 2020.

[4] Jia-Qi Yang, Xiang Li, Shuguang Han, Tao Zhuang, De-Chuan Zhan, Xiaoyi Zeng, and Bin Tong. Capturing Delayed Feedback in Conversion Rate Prediction via Elapsed-Time Sampling. AAAI, 2021.

[5] Yifan Hu, Yehuda Koren, and Chris Volinsky. Collaborative Filtering for Implicit Feedback Datasets. ICDM, 2008.

[6] Rong Pan, Yunhong Zhou, Bin Cao, Nathan N. Liu, Rajan Lukose, Martin Scholz, and Qiang Yang. One-Class Collaborative Filtering. ICDM, 2008.

[7] Xiao Ma, Liqin Zhao, Guan Huang, Zhi Wang, Zelin Hu, Xiaoqiang Zhu, and Kun Gai. Entire Space Multi-Task Model: An Effective Approach for Estimating Post-Click Conversion Rate. SIGIR, 2018.

AuthorsOpenReview ↗

Response to Reviewer Sou8, Part 2/5

W2: Theoretical justification for Retrieval Rank and the RGC/FCR design.

GRPO-style methods compute relative advantages independently within each rollout group, so an advantage mainly captures a candidate's reward difference relative to its own group and cannot retain comparisons across groups. For example, even if all candidates in one group have good recommendation value, as long as rewards differ within the group, the relatively weaker candidates still receive negative advantages; meanwhile, in another group of lower overall quality, the relatively better candidates may receive positive advantages. In recommendation, where positive feedback is sparse [5–7], this means that some potentially valuable recommendations are suppressed merely because of within-group comparison. Recommendation is essentially a ranking task, and our reward is computed directly from the Retrieval Rank. The Rank is the position of the observed target item in the full-catalog retrieval list induced by the candidate and is comparable across the entire batch, which supplies exactly the information that within-group advantages lack. Figure 5 supports this empirically: after matching on the observed advantage magnitude, better-ranked negative-advantage samples still have a higher conflict rate, with a pooled difference of 9.1 percentage points and a 95% CI of [4.5, 14.7], indicating that Rank carries information beyond advantage magnitude.

Sign Conflict provides an observable diagnosis of how sensitive a negative update is to changes in feedback: a conflicting negative-advantage sample suppresses a recommendation in a direction inconsistent with subsequent feedback and is therefore more likely to be an unreliable negative update, whereas a non-conflicting sample is more likely to provide a valid negative learning signal. Applying attenuation αᵢ to sample i scales its advantage coefficient by 1 − αᵢ, and, as derived under W7, its policy-gradient contribution is scaled by exactly the same factor. RGC measures the attenuation coverage of conflicting negative-advantage mass, while FCR measures the degree of attenuation applied to non-conflicting negative-advantage mass. Maximizing RGC subject to FCR ≤ ε therefore weakens as much of the likely unreliable suppression as possible while controlling the impact on non-conflicting negative updates, analogous to maximizing the detection rate at a fixed false-alarm rate. Optimizing RGC/FCR does not theoretically guarantee downstream gains; we use it as a calibration objective directly tied to the reliability of negative updates and verify it through both offline coverage experiments and downstream experiments. The results in Table 2 of the main paper are consistent with this mechanism: at a 10% budget, undifferentiated Uniform attenuation underperforms GRPO in both domains, while RCNS, which achieves higher RGC, obtains the best downstream results among all attenuation variants on every metric. RCNS therefore brings recommendation-specific, cross-group ranking information into GRPO-style methods and uses it to attenuate negative updates selectively under a risk constraint.

References

[5] Yifan Hu, Yehuda Koren, and Chris Volinsky. Collaborative Filtering for Implicit Feedback Datasets. ICDM, 2008.

[6] Rong Pan, Yunhong Zhou, Bin Cao, Nathan N. Liu, Rajan Lukose, Martin Scholz, and Qiang Yang. One-Class Collaborative Filtering. ICDM, 2008.

[7] Xiao Ma, Liqin Zhao, Guan Huang, Zhi Wang, Zelin Hu, Xiaoqiang Zhu, and Kun Gai. Entire Space Multi-Task Model: An Effective Approach for Estimating Post-Click Conversion Rate. SIGIR, 2018.

AuthorsOpenReview ↗

Response to Reviewer Sou8, Part 3/5

W3 / W8: Missing confidence- or uncertainty-weighted baselines; it is hard to separate reliability modeling from mere reweighting.

RCNS is implemented as negative-advantage reweighting, and our contribution lies in answering two questions: what information the reweighting should be based on, and how to constrain its side effects. Retrieval Rank is not the only signal that can reflect update reliability; we chose it because it is directly tied to the recommendation reward, is readily available during training, and requires no additional uncertainty estimator.

Advantage magnitude and reward margin. In GRPO, Aᵢ is the within-group reward margin rᵢ − μₓ normalized by the within-group standard deviation. The Figure 5 analysis matches negative-advantage samples one-to-one on |Aᵢ| and still finds a higher conflict rate for better-ranked samples, so Rank provides information beyond the normalized advantage magnitude; this analysis does not separately control for unnormalized reward margins across groups.

Model confidence and uncertainty. We constructed two confidence-based negative-advantage weighting baselines. For each generated output, we use the average log-probability of its valid tokens as the confidence score, rank only the negative-advantage samples within the batch by confidence percentile, and consider two directions: low-confidence-first attenuation, where negative updates on which the model is less certain are attenuated more, and high-confidence-first attenuation, where negative updates on which the model is more certain are attenuated more. Both baselines use the same attenuation function as RCNS and differ only in the priority signal.

We first compare offline allocation efficiency. All rows use the fixed seed-42 diagnostic pool and the Table 15 pipeline; under a 10% validation FCR budget, each priority signal calibrates its own exponent p, and Uniform calibrates a constant attenuation rate. The RCNS row reproduces the pretrained reference in Table 15, and Uniform's Precision equals the test conflict mass in Figure 3. Test results in %:

DatasetMethodConflict massRGCFCRPrecision
C&VUniform4.6410.0010.004.64
C&VLow-confidence-first4.647.1310.113.32
C&VHigh-confidence-first4.6415.6510.007.08
C&VRCNS4.6417.039.318.17
VGUniform10.5410.0010.0010.54
VGLow-confidence-first10.543.0310.043.43
VGHigh-confidence-first10.5416.7810.1416.30
VGRCNS10.5416.0510.5015.25
M&TVUniform5.9810.0010.005.98
M&TVLow-confidence-first5.9811.459.966.81
M&TVHigh-confidence-first5.9813.0610.347.43
M&TVRCNS5.9815.569.849.13

Under the same validation FCR budget, low-confidence-first attenuation yields lower RGC than Uniform on C&V and VG and is only slightly higher on M&TV, while high-confidence-first attenuation exceeds Uniform in all three domains. Retrieval Rank achieves the highest RGC on C&V and M&TV and is close to high-confidence-first on VG. We further compared downstream performance on Movies & TV. Each method's attenuation strength is calibrated separately under a 10% validation FCR budget, all methods use the same training budget of 2 epochs and 140 steps, and all are evaluated on the same 15,693 test records, in units of ×10³:

MethodHR@5NDCG@5HR@10NDCG@10
GRPO8.30 ± 0.695.53 ± 0.6411.76 ± 1.596.64 ± 0.91
Low-confidence-first6.96 ± 0.754.67 ± 0.6310.68 ± 1.265.85 ± 0.78
High-confidence-first9.19 ± 1.536.09 ± 0.9714.90 ± 2.307.90 ± 1.20
RCNS9.93 ± 2.046.61 ± 1.4714.49 ± 2.588.06 ± 1.62
Low-confidence-first vs. GRPO−16.13%−15.67%−9.21%−11.93%
High-confidence-first vs. GRPO+10.75%+9.95%+26.65%+18.93%
RCNS vs. GRPO+19.66%+19.35%+23.19%+21.29%

On Movies & TV, low-confidence-first attenuation performs close to Uniform in Table 2 and below GRPO on all four metrics. High-confidence-first attenuation outperforms GRPO on all four metrics, and RCNS achieves the best results on HR@5, NDCG@5, and NDCG@10, while high-confidence-first attenuation is slightly higher on HR@10. These results suggest that, within the same risk-calibrated framework, attenuation benefits depend on the priority signal. Retrieval Rank achieves the highest RGC in two of the three domains and the best downstream performance on three of the four metrics evaluated on Movies & TV.

Reviewer_ACG5OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

GRPO-style RL suppresses recommendations with negative group-relative advantages, but training-time user feedback is incomplete. The authors show that 5.2%–13.1% of negative-advantage recommendations flip to positive advantages under held-out feedback. They find these conflicts concentrate among better-ranked negative recommendations, and propose RCNS that rank-based priorities + continuous, sign-preserving attenuation, with strength calibrated once per domain on validation feedback under an empirical risk budget. RCNS doubles offline conflict coverage vs. uniform attenuation at the same risk budget and improves HR@5 over GRPO by 5.0%–19.7% across three domains.

Critical strengths

1: A well-diagnosed problem with an observable training-time cue.The paper identifies a real and previously underexamined failure mode of GRPO-style training in recommendation: negative group-relative advantages can be unreliable under incomplete feedback. The diagnostic is clean — fixing recommendations and rollout groups while swapping evaluation feedback reveals recurring sign conflicts across domains and backbones. 2: Principled method design with unusually rigorous and honest evaluation.RCNS is well-motivated and disciplined: attenuation is bounded and sign-preserving, strength is calibrated once per domain under an explicit empirical risk budget (FCR), and no extra feedback is needed at training time. The evaluation uses five independent seeds with paired per-seed differences, bootstrap confidence intervals, calibration-transfer checks across training passes, and extensive robustness ablations.

Critical weaknesses
  1. All findings are derived from only three domains of Amazon Reviews'23. Because these domains share the same platform, input format, and splitting protocol, they do not constitute independent evidence of generality. It therefore remains unverified whether the observed sign-conflict rates and the advantage of rank-based over uniform attenuation generalize to datasets with different feedback density or exposure processes.
  2. The baselines are narrow and partially self-implemented. The RL comparison covers only GRPO and two author-reimplemented HPO variants. Rec-R1 (Lin et al., TMLR 2025) is not compared, and recent GRPO refinements targeting negative-advantage treatment, such as DAPO (Yu et al., 2025), are neither discussed nor evaluated.
Questions for authors

Q1:The reward and the downstream evaluation both use Qwen3-Embedding-0.6B. Have you verified that RCNS's gains persist when the final evaluation uses a different encoder are the improvements genuine recommendation quality rather than overfitting to a specific embedding space? Q2:In Appendix G, most 95% bootstrap confidence intervals for the paired RCNS–GRPO differences cross zero (e.g., C&V ΔHR@5: +0.264 [−0.125, 0.640]). Do you have additional evidence likes more seeds, larger models to establish statistical significance for the claimed improvements?

5 comments
AuthorsOpenReview ↗

Response to Reviewer ACG5, Part 1/4

We thank the reviewer for recognizing our motivation, method design, and the rigor of our evaluation. To address the reviewer's concerns, we conducted the following additional experiments: a new Yelp dataset, plug-in experiments on Rec-R1 and DAPO, experiments with alternative evaluators and encoders, and 20 additional paired training runs per domain. In each of these settings, RCNS achieves higher mean downstream performance than the corresponding baseline, and the 20 paired runs provide stronger statistical support. Unless otherwise stated, all downstream experiments follow the main paper and use five independent training seeds. Recommendation metrics follow the format of Table 1 in the main paper and are reported as mean ± sample SD in units of ×10³.

W1: All findings are based on three Amazon domains.

Feedback noise in recommender systems has long been recognized. In industrial recommendation, a successful conversion may require multiple exposures [1], so the absence of interaction after early exposures is not necessarily reliable evidence of negative preference. Extensive prior work has addressed this issue through exposure modeling and delayed-feedback modeling [2–4]. In reinforcement learning for generative recommendation, however, the model first generates a group of candidates and then scores them and computes relative advantages based on the currently observed feedback. Incomplete feedback can therefore be converted into negative policy updates through group-relative advantages, wrongly suppressing recommendations with potential value. To the best of our knowledge, this paper is the first to systematically study the sign instability of negative advantages in generative-recommendation RL under incomplete user feedback, which we formalize as Advantage Sign Conflict. Since incomplete feedback is pervasive in recommendation, this problem is not specific to any particular platform or dataset, but a general optimization challenge for RL-based generative recommendation.

We agree with the reviewer that the three Amazon domains come from the same platform and that cross-platform evidence is necessary. We therefore added the Yelp dataset. Yelp is a local-services platform whose business scenario, item types, and interaction distribution all differ from Amazon's e-commerce setting. Following the same diagnostic procedure as Figure 3, the table below reports the share of negative-advantage mass that undergoes a sign conflict in the Train → Test setting, in %:

CDs & VinylVideo GamesMovies & TVYelp
4.610.56.015.1

Sign Conflict also occurs on Yelp, with a higher conflict share. We further compared attenuation strategies under the same validation FCR budget of 10%, where Linear refers to the batch-relative linear attenuation in Appendix D.2. Test results are reported in %:

MethodRGCFCRPrecision
RCNS10.349.7715.85
Linear9.649.9214.74
Uniform10.0010.0015.10
Random10.4210.4415.06

Under the same validation FCR budget, RCNS improves RGC by 3.43% and 7.31% relative to Uniform and Linear, respectively, while achieving the lowest FCR and the highest Precision. For downstream recommendation performance, we trained GRPO and RCNS on Yelp:

MethodHR@5NDCG@5HR@10NDCG@10
GRPO9.59 ± 0.817.20 ± 0.7912.17 ± 0.608.03 ± 0.70
RCNS10.47 ± 1.537.76 ± 1.1313.74 ± 2.108.82 ± 1.28
Relative gain+9.13%+7.83%+12.93%+9.80%

RCNS outperforms GRPO on all four metrics with relative gains of 7.83%–12.93%, showing that the downstream benefits of RCNS also hold on an independent platform.

References

[1] Xuhui Shao and Lexin Li. Data-driven Multi-touch Attribution Models. KDD, 2011.

[2] Dawen Liang, Laurent Charlin, James McInerney, and David M. Blei. Modeling User Exposure in Recommendation. WWW, 2016.

[3] Yuta Saito. Unbiased Pairwise Learning from Biased Implicit Feedback. ICTIR, 2020.

[4] Jia-Qi Yang, Xiang Li, Shuguang Han, Tao Zhuang, De-Chuan Zhan, Xiaoyi Zeng, and Bin Tong. Capturing Delayed Feedback in Conversion Rate Prediction via Elapsed-Time Sampling. AAAI, 2021.

AuthorsOpenReview ↗

Response to Reviewer ACG5, Part 2/4

W2: More baselines are needed, such as Rec-R1 and DAPO.

We thank the reviewer for the suggestion. RCNS targets a limitation inherent to group-relative advantages in GRPO-style methods: whenever rewards differ within a group, the within-group comparison assigns negative advantages to some candidates, even if those candidates have potential recommendation value. This limitation does not depend on the specific backbone or reward design and applies to any generative recommendation method trained with group-relative advantages. RCNS only acts on the update of negative advantages, obtains the required ranking information directly from reward computation, and does not modify the reward, rollout construction, or model architecture. It can therefore be transferred to such methods as a plug-in.

Regarding HPO, we implemented it strictly following the method design of the original paper and compared it fairly under the same data, splits, and training settings. Our implementation will be released together with our code.

Rec-R1 [5]. We use Rec-R1 as the backbone, keep its full training pipeline unchanged, and plug RCNS in only at the negative-advantage update. Results on Movies & TV:

MethodHR@5NDCG@5HR@10NDCG@10
Rec-R110.51 ± 1.176.53 ± 0.5517.78 ± 1.358.85 ± 0.60
Rec-R1 + RCNS12.16 ± 0.597.46 ± 0.1820.01 ± 0.849.99 ± 0.29
Relative gain+15.61%+14.25%+12.54%+12.88%

DAPO [6]. DAPO improves training stability and efficiency by modifying sampling, clipping, and the computation of the policy gradient, whereas RCNS focuses on the reliability of negative-advantage updates. The two target different optimization problems and are complementary. To verify this composability, we integrated RCNS into DAPO with all other settings identical to the main experiments. Results on Movies & TV:

MethodHR@5NDCG@5HR@10NDCG@10
DAPO10.28 ± 1.786.77 ± 1.2416.21 ± 2.468.68 ± 1.43
DAPO + RCNS11.70 ± 1.757.70 ± 1.2417.58 ± 2.149.60 ± 1.34
Relative gain+13.86%+13.79%+8.45%+10.58%

References

[5] Jiacheng Lin, Tian Wang, and Kun Qian. Rec-R1: Bridging Generative Large Language Models and User-Centric Recommendation Systems via Reinforcement Learning. TMLR, 2025.

[6] Qiying Yu et al. DAPO: An Open-Source LLM Reinforcement Learning System at Scale. NeurIPS, 2025.

AuthorsOpenReview ↗

Response to Reviewer ACG5, Part 3/4

Q1: Do the gains depend on a specific embedding space?

We thank the reviewer for this important question. To further verify whether the gains of RCNS depend on Qwen3-Embedding-0.6B, the encoder used for the training reward, we conducted additional experiments from two angles: evaluation with independent evaluators and retraining in a different representation space.

First, we directly evaluated the original models with independent retrievers. We kept the GRPO and RCNS models trained with the Qwen3-Embedding reward, as well as their generated outputs, unchanged, and replaced only the retriever used in the final evaluation with BGE-M3 [7] and BM25. The former uses an independent dense semantic representation space, while the latter relies entirely on lexical matching and does not depend on the embedding representation used during training. Results across all three Amazon domains:

DatasetEvaluatorMethodHR@5NDCG@5HR@10NDCG@10
C&VBGE-M3GRPO3.30 ± 0.392.23 ± 0.284.88 ± 0.542.73 ± 0.31
RCNS3.58 ± 0.372.42 ± 0.344.99 ± 0.502.88 ± 0.39
BM25GRPO2.15 ± 0.321.43 ± 0.273.86 ± 0.361.98 ± 0.28
RCNS2.55 ± 0.271.65 ± 0.153.94 ± 0.352.10 ± 0.15
M&TVBGE-M3GRPO4.49 ± 0.443.09 ± 0.476.51 ± 0.713.73 ± 0.56
RCNS5.70 ± 0.783.98 ± 0.697.76 ± 1.194.63 ± 0.81
BM25GRPO7.20 ± 0.944.70 ± 0.4911.06 ± 1.405.93 ± 0.64
RCNS8.60 ± 1.495.65 ± 1.0713.34 ± 1.527.15 ± 1.07
VGBGE-M3GRPO10.55 ± 0.886.79 ± 0.6417.11 ± 1.058.89 ± 0.67
RCNS10.91 ± 1.166.97 ± 0.8218.15 ± 1.729.29 ± 1.00
BM25GRPO9.71 ± 0.726.02 ± 0.5216.24 ± 0.958.11 ± 0.59
RCNS10.15 ± 1.196.33 ± 0.8017.09 ± 1.658.55 ± 0.96

Under both independent retrievers, BGE-M3 and BM25, RCNS achieves higher mean performance in all 24 comparisons across the three domains, with relative gains of 1.94%–28.74%. In particular, BM25 relies entirely on lexical matching and uses a retrieval mechanism different from the dense semantic space used in training. These results show that the gains of RCNS carry over to independent semantic and lexical retrieval evaluation beyond the Qwen3-Embedding space used by the training reward.

In addition, we verified the effectiveness of RCNS in an entirely different representation and reward space. In our framework, the encoder participates in both the training reward computation and the final item retrieval. We therefore replaced the encoder in both stages with BGE-M3, using BAAI/bge-m3 with CLS pooling, recalibrated p⋆ under the new reward with the same 10% validation FCR budget, kept all other settings unchanged, and retrained GRPO and RCNS on Movies & TV:

MethodHR@5NDCG@5HR@10NDCG@10
GRPO9.20 ± 1.196.67 ± 0.9713.71 ± 1.528.09 ± 1.06
RCNS10.41 ± 2.287.61 ± 1.7515.10 ± 3.099.10 ± 1.98
Relative gain+13.16%+14.20%+10.13%+12.48%

With BGE-M3 used for both training and evaluation, RCNS achieves higher mean performance on all four metrics, with relative gains of 10.13%–14.20%. Taken together, the two sets of experiments show that the gains of RCNS hold across different retrieval representations and retrieval mechanisms, as well as across different reward constructions and training settings, and are not tied to a specific embedding space.

References

[7] Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. BGE M3-Embedding: Multi-Lingual, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation. Findings of ACL, 2024.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.