acceptodds
Under review as a conference paper at ICLR 2027

Don’t Let Bandit Feedback Pull Continual LLM-Recommender Updates Off Target

Abstract

Generative LLM-based recommenders (LLM-Rec) require continual post-deployment updates, yet deployment logs provide only policy-shaped contextual bandit feedback: outcomes are observed solely for items exposed by a prior serving policy, inducing exposure bias and yielding partial, asymmetric signals consisting of relatively reliable positive responses and ambiguous no-responses. We propose an Anchored Bandit Policy Optimization (ABPO) framework for continual LLM-Rec updates that combines group-relative policy optimization (GRPO) with explicit treatment of exposure bias and feedback ambiguity. Specifically, we insert the exposed recommendation as a logged anchor into each GRPO rollout group, so that group-relative normalization is calibrated against the action actually exposed by the prior policy rather than against newly sampled rollouts alone. Because both positive- and no-responses are observed only through prior-policy exposure, we apply self-normalized inverse propensity scoring to the fixed anchor for both feedback types to correct for policy mismatch. At the same time, we treat the two feedback types asymmetrically in reliability: positive responses provide relatively direct endorsement signals, whereas no-responses remain ambiguous because they may reflect either true disinterest or unobserved external factors. To avoid overly aggressive updates from ambiguous no-responses, we temper their penalties with self-certainty, using the model's output-token confidence as a verifier-free reliability signal. Across five domains from Amazon Reviews and MovieLens, our method yields consistent post-update gains in recommendation accuracy while mitigating prior-policy-induced exposure bias more effectively than prior baselines. A real-world online A/B test further shows that our method achieves strong CTR and gradual improvement over successive production updates, demonstrating its practical effectiveness in production.

Primary area:
generative models
Keywords:
Generative Recommendation, Bandit Feedback, Continual Learning

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_J119OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

Framework for updating LLM-based recommenders using post-deployment feedback + anchoring updates to historically exposed items to correct for exposure bias. It treats positive and no-response feedback with self-normalized inverse propensity scoring. It treats positive feedback (clicks) differently from no-response feedback (no clicks), using the model's confidence to avoid over-penalizing unclear signals.

This framework showed good results on online and offline experiments.

Critical strengths

The authors spent quite some time working on the validation (offline and online), and they added several baselines + datasets. This helps build trust on their results and method.

Critical weaknesses

Their statistical analysis could use some more rigour. In particular, AB tests often require power analysis to ensure robustness of the statistical test - but there was no mention on how any approach might impact (positively or negatively).

In bandits work, propensity weighting to correct for selection biases is also quite common. They do make some comparisons with other correction biases, but it would be valuable to also have a more close comparison with a very well-known standard methods.

Questions for authors

The authors mentioned 'diversity' - but it is unclear its overall role and impact. Can't this be something learned? Or tuned? It looks like one of the goals is to have more 'diversity' (not recommend same item over and over), but one might be able to measure and include that on framework?

Considering how strong the 'no update' baseline was, and that naive contextual bandit updates might harm performance. Even if I add the proposed approach, the gains are very marginal at the cost of a lot of complexity to maintain the framework. Is there any additional motivation to an user to adopt this framework? Does it converge faster? Require less samples to obtain enough statistical power?

3 comments
AuthorsOpenReview ↗

Response to Reviewer J119

Thank you for pressing on rigor and on practical value. We have revised the manuscript to address the reviewers' comments and highlighted the changes in red. We address each point below.


W1 (power). Addressed in Global Response [G3].

  • Users were randomized, so we now test at the user level with user-clustered standard errors (design effect 3.1; §4.5, App. D).
  • At 80% power, the minimum detectable CTR difference against ML-Retrain is about 0.41–0.44 pp for a single month and about 0.25 pp for the three months pooled. A single month is therefore underpowered for an effect of this size, which is why February alone is not significant (z=0.68z=0.68).
  • Our claim is the pooled one: +0.34 pp (7.44% vs. 7.10%; clustered z=3.83z=3.83, 95% CI [+0.17, +0.51]), which is above the pooled MDE. We no longer draw conclusions from the month-to-month changes.

W2 (standard propensity baselines). Addressed in Global Response [G4].

  • We ran an IPS policy gradient and a doubly-robust (DR) estimator on the same logs (Table R6), and ABPO is best on all three domains we tested. IPS is high-variance here because a generative policy's item propensities are peaked (max weight 13.7), and it falls below No-Update on Grocery and CDs; DR is stronger but still trails ABPO by 1.7–4.3 pp.
  • They do not subsume ABPO because they act on a different quantity. IPS and DR reweight a pointwise return, whereas GRPO learns from a group-relative advantage, and the anchor works inside that group normalization ([G1]), which neither estimator touches.

Q1. (Clarification) Diversity Is a Reported Safety Outcome, Not a Tuned Objective.

Takeaway. Updating on bandit logs tends to concentrate exposure: positive-only and hard-negative updates cut diversity by 23–48% (Fig. 1), while ABPO stays close to the starting policy (0% / −5% on Grocery / MovieLens; Table R4). We therefore report diversity as a check that an update has not narrowed what users see, rather than as something we optimize.

  • Table 2 now includes No-Update, so each update rule can be compared with the starting policy. The competing rules often fall below it (35 of 72 Div@1/Div@5 entries), while ABPO has the highest Div@5 in 11 of 12 settings. Coverage and long-tail share show the same pattern ([G5] table): ABPO keeps coverage near the starting policy (11.4% vs. 12.1%) and raises the long-tail share (18.2% vs. 16.8%), whereas every competing rule lowers both.
  • Diversity can also be optimized directly: adding a diversity reward (λdiv=0.1\lambda_{\text{div}}=0.1) raises Div@5 by 1.1 but costs 0.5 pp of HR@5. We keep it out of the objective so that the update rule's own effect on exposure stays measurable.

Q2. (Clarification) The Gain Is a Safe Update, Not a Marginal One.

Takeaway. The comparison that matters in deployment is updating versus not updating. On the same logs, the other update rules fall below No-Update in 63 of 72 HR@5 cells, so a naive update often makes the deployed model worse. ABPO is best in all 24 backbone–domain settings on HR@5 and NDCG@5 (17 of 24 on HR@1), with a median HR@5 gain of +15.3% over No-Update. The gain is smallest on the strongest backbone, G²RPO (median +3.2%), and online it is +0.34 pp CTR, a 4.7% relative improvement.

  • Complexity. ABPO adds one group member that receives no gradient, one weight normalization, and one scalar reward to a standard GRPO update (Eqs. 4–7). Training time increases by 6%, and serving is unchanged.
  • Sample efficiency. With 25% / 50% / 100% of the logs, ABPO reaches 19.2 / 20.8 / 21.93 HR@5 on Health, versus 13.9 / 15.2 / 16.1 for vanilla GRPO. ABPO trained on a quarter of the logs already beats vanilla trained on all of them.
  • Convergence. We do not claim faster convergence per training step; the sample-efficiency result above is the efficiency claim we can support.
  • Power. The online gain (+0.34 pp) is above the pooled minimum detectable effect (about 0.25 pp); see W1.

Thanks for all the comments. Based on the authors feedback + global answer, I'll update my score from 2 to 3.

Thank you so much for your kind follow-up and for updating your score! We really appreciate your time and valuable feedback.

Reviewer_apHmOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies exposure bias and feedback uncertainty in the continual updating of generative LLM-based recommenders (LLM-Rec). The authors propose Anchored Bandit Policy Optimization (ABPO), which inserts the item actually exposed by the previous serving policy as an anchor into GRPO's group-relative normalization and combines self-normalized inverse propensity scoring (SNIPS) with a self-certainty reward to treat positive and no-response feedback asymmetrically. Experiments are conducted offline on several Amazon Reviews subsets and MovieLens, and the method is further evaluated in an online A/B test.

Critical strengths
  1. The problem is well chosen and practically important. The paper targets continual post-deployment updating of recommenders and accounts both for the exposure bias induced by the serving policy and for the fact that a no-response cannot be equated with negative preference. Both issues are pervasive in industrial recommender systems, and the motivation is well grounded.

  2. The method design fits the structure of the problem. ABPO injects the actually exposed item into GRPO's group-relative reward normalization, so that updates are calibrated against real logged feedback; it then modulates the influence of the two feedback types through propensity weighting and a self-certainty reward. The framework integrates naturally with existing RL pipelines for generative recommendation and appears readily extensible.

  3. Experimental coverage is broad and includes online evidence. The experiments span multiple recommendation domains and four LLM-Rec backbones, and ablation studies isolate the contribution of each component. Compared with work that reports offline metrics only, the paper additionally provides three consecutive months of online CTR results, which is a valuable complement to the offline evidence.

Critical weaknesses
  1. The theoretical justification for the exposure-bias correction needs to be strengthened. The SNIPS weight is applied to the logged anchor's contribution to the group-relative reward mean rather than to a complete off-policy return estimate, and the appendix itself notes that this does not guarantee an unbiased GRPO objective. The current analysis is therefore insufficient to establish that ABPO effectively removes the exposure bias induced by the historical policy. The authors should distinguish more explicitly between "modulating the influence of historical exposure" and "correcting off-policy bias," and examine the relationship between the two with additional experiments.

  2. The offline feedback simulation deviates substantially from real user behavior. In the offline construction, the true next interacted item is placed in the candidate set, and positive versus no-response feedback is defined by whether the historical policy selected that item. This design is convenient for controlled experimentation, but real clicks and non-responses are also shaped by presentation position, user attention, and interest drift. As constructed, the feedback ambiguity that the paper centers on is not actually exercised. The authors should incorporate data with genuine exposure information, or design experiments with false negatives and missing responses at varying rates.

  3. Users are randomized at the user level, yet the significance tests in the appendix are two-proportion tests at the impression level, which may not account for correlation among repeated impressions from the same user. In addition, the February CTR difference between ABPO and ML-Retrain is not significant, and the later month-over-month gains may be confounded by changes in user composition and in the external environment. The authors should use user-level clustered tests or bootstrap, and report the traffic allocation per arm, the duration of the experiment, and the update schedule.

  4. It remains unclear whether the self-certainty reward actually reflects feedback reliability. The authors use the concentration of the model's output distribution as an auxiliary reward for no-response cases, but a model being confident about an item does not imply that the recommendation matches the user's true preference; the design may also reinforce predictions about which the model is already confident. The authors should analyze the correlation between confidence and true preference, and compare directly against entropy regularization, a fixed weight, or other uncertainty estimates.

Questions for authors

See Weaknesses

3 comments
AuthorsOpenReview ↗

Response to Reviewer apHm

Thank you for a review whose four points map directly onto what we changed. We have updated the manuscript in response to the reviews and marked the revisions in red. We address each point below.


W1 (modulating vs correcting). Addressed in Global Response [G4] and [G1].

  • Your distinction is adopted verbatim: the weight transports the anchor's contribution to the current exposure distribution (Ea∼eold[f(a)∣y]\mathbb E_{a\sim e_{\text{old}}}[f(a)\mid y]); it does not make the objective unbiased, and §3.3 says so.
  • Full off-policy baselines (IPS, DR) trail ABPO on all three domains; [G1] shows why: the anchor acts on the group normalization, which pointwise estimators do not touch.

W2 (simulation realism). Addressed in Global Response [G2].

  • Position, attention and drift are absent, agreed. We show the selection effect the simulation does contain (Fig. 1: −13% / −20% with exact labels; gone under random selection) and inject false negatives at 10 / 30 / 50%, an examination model and latent positives.
  • The online deployment is the genuine-exposure setting; the simulation is kept for the controlled protocols.

W3 (online statistics). Addressed in Global Response [G3].

  • User-clustered tests: February n.s. (z=0.68z=0.68), as you noted; March / April / pooled significant (2.65 / 3.34 / 3.83).
  • Agreed on confounding: we withdraw the month-over-month reading. The claim is the pooled advantage over the concurrent ML-Retrain arm, which is exposed to the same seasonality and user-composition changes; since the difference-in-differences is borderline (z≈2.0z \approx 2.0), we do not claim the effect grows over time. Impression shares 14 / 18 / 25 / 43% (Popularity / ML-Retrain / GRPO-Update / ABPO), 3 months, weekly updates.

W4 (self-certainty). Addressed in Global Response [G5].

  • Confidence vs preference: AUROC 0.68 / 0.71, modest by design (SC tempers, it does not decide).
  • Direct comparisons: entropy reward 20.6, fixed negative scaling 20.9, confidence-weighted logged item 20.7 vs SC 21.93 (Health HR@5).
  • Long-tail share and coverage stay at the No-Update level under ABPO while every competing rule lowers them.

Thank you for the rebuttal. The authors have addressed my concerns, and I will maintain my original score.

Thank you for taking the time to consider our response. We are glad it addressed your concerns, and we appreciate your positive assessment and thoughtful feedback.

Reviewer_q3G2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The authors examine updating a deployed LLM recommender (LLM-Rec) from its own logs, where one can only see whether an item was clicked or not. The proposed framework, Anchored Bandit Policy Optimization (ABPO), builds on continual LLM-Rec updates that combine group-relative policy optimization (GRPO) with explicit treatment of exposure bias and feedback ambiguity. In this context, the logged item is put into each rollout group, but only to compute the group's average reward as a baseline. The anchor's weight in the baseline is an importance ratio between the new and old policies, normalized separately for clicks and no-clicks. For no-click examples, rollouts also get a reward for how confident the model is in its output. Offline, the logs are simulated from 5 Amazon domains and MovieLens with 4 recommender backbones. A logged item counts as a click only if it equals the user's true next item. The proposed method beats DEALRec, RL-Rec, and UL-Rec on HR/NDCG and a popularity-based diversity score. There's also a 3-month online A/B test showing CTR changing from 7.23% to 7.62%.

Critical strengths
  • Most previous LLM-Rec works ignore updating LLM recommenders from their own biased logs. This paper explains why this issue is important for GRPO-style training. It shows that rewarding matches to clicked items pulls the policy toward old exposures, while penalizing no-clicks quickly pushes those items out of the model's outputs.
  • The offline evaluations demonstrate that naive feedback-driven updates can actually hurt the results.
  • The three-month online A/B test, which is uncommon in this area, is interesting and informative.
Critical weaknesses
  • Theorem 1 and Proposition 1 show that the anchor moves the baseline by the same amount, ω/(ω+G−1) ≈ 1/16, for every sampled rollout. A uniform baseline shift can't lower the advantage of rollouts that copy the logged action. A rollout that matches a positive anchor still gets the highest advantage in the group. When no rollout matches, the anchor gives the group nonzero reward variance, so the batch still produces a learning signal instead of nothing. Moreover, SNIPS only rescales a stop-gradient weight on a 1/16 term. Therefore, it is not evident that it can drive the gains reported in the ablation study. Without error bars, the D-vs-C and E-vs-C differences could be noise.
  • DEALRec is a data-pruning method, and RL-Rec comes from query suggestion. Standard off-policy learning baselines are missing. Online, traffic was allocated by prior performance, so the arms do not have equal shares. Moreover, the significance test runs at the impression level (App. D) even though the main text says user level, which inflates the z-scores.
  • In Appendix A, y=1 exactly when the logged item is the true next item, and y=0 means the logged item was a random item the user never interacted with. Therefore, a no-click is basically a clean negative, and a click is the ground-truth label. The ambiguity in no-clicks that motivates self-certainty never appears in the data. In Fig. 2, once matching a positive means matching the ground truth, a rising match rate shows the model getting more accurate, not biased. So the offline results show the method works as a reward-shaping trick for supervised next-item RL. They don't show that it handles exposure bias.
Questions for authors
  • The introduction says SNIPS is applied to both feedback types, but ablation (D) says positives only. Which one is right?
  • Under the simulation conditions of M=200 and Softmax over the SFT model, what fraction of the logged data has y=1?
  • Does the gain come from the anchor or from having a non-degenerate group std?
  • Do you have any offline setting where no-click is actually ambiguous, e.g., clicks simulated with position bias or an examination probability?
  • What happens if all G rollouts are sampled with no anchor, but the logged item's reward is still used?
  • What are λ_sc, τ, and δ, and how sensitive are the results to them?
  • In the online test, do you consider one update per month? Can you provide user-level clustered CIs?
2 comments
AuthorsOpenReview ↗

Response to Reviewer q3G2 (1/2)

Thank you for a precise review; your derivation for the mean channel is correct and it reshaped our analysis. We have revised the manuscript to address the reviewers' comments and highlighted the changes in red. We address each point below.


W1 (anchor, SNIPS, error bars). Addressed in Global Response [G1].

  • As you mentioned, the anchor shifts the mean by the same amount for every rollout, so the mean alone cannot damp copying. The damping comes from the std, which the anchor also enters: the copy advantage (1−p′)/p′\sqrt{(1-p')/p'} falls from 3.74 to 2.65 at ω=1\omega=1 (§3.5 Remark), and we measure 2.9 → 2.3 during training.
  • On the "1/16" point: w^\hat w sets the anchor weight ω\omega, and ω\omega changes the std that all 15 advantages are divided by, so its effect is not a single 1/16 term. When no rollout copies (m=0m=0), every rollout gets a small negative advantage (−0.26-0.26 at ω=1\omega=1), a mild push toward an item with a positive response that accounts for at most 9% of the expected push toward alog⁡a^{\log}.
  • Error bars: three seeds, std ≤ 0.9 pp. The exposure weight adds +1.7 / +2.9 pp (C→D, Health / CDs; Table 3), which is at least 2 std. Self-certainty adds only +0.6 / +0.8 pp on the clean logs (C→E), as expected without ambiguity, and +2.9 pp when 50% of positives are flipped (Table R3).

W2 (baselines). Addressed in Global Response [G4].

  • We ran an IPS policy gradient and a doubly-robust estimator on the same logs, and ABPO is best on all three domains tested (Grocery / CDs / Health; Table R6). IPS is high-variance with peaked item propensities, and DR trails ABPO by 1.7–4.3 pp.

W2 (online). Addressed in Global Response [G3].

  • The original tests treated impressions as independent, although users were the unit of randomization. With user-clustered standard errors (design effect 3.1; §4.5, App. D), ABPO's gain over ML-Retrain is significant in March, April, and pooled (zz = 2.65 / 3.34 / 3.83) but not in February alone (zz = 0.68).
  • Impression shares were 14 / 18 / 25 / 43% (Popularity / ML-Retrain / GRPO-Update / ABPO) and constant across months. Because users were hash-randomized, unequal shares affect power, not bias; a traffic-matched subsample gives 7.41% vs. 7.10% (p=0.003p=0.003).

W3 (simulation). Addressed in Global Response [G2].

  • We agree that the labels are exact. The bias in our logs is which contexts get a positive at all, which depends on what the logging policy happened to expose. Choosing the same number of positives at random removes most of the Fig. 1 drop (HR@5 −13% → −2%, Grocery), and ABPO on the same e0e_0 logs turns it into a +14% gain (Table R4). §4.1 now states this scope.
  • Fig. 2 plots a matching rate on the training logs, i.e., the share of rollouts that reproduce the logged item, not accuracy. Vanilla's matching rate climbs to ~60% while its held-out HR@5 falls below No-Update on CDs (30.9 vs. 32.59; Table 3), so it memorizes exposures rather than learning preferences. §2.1 and the Fig. 2 caption now define the matching rate.
  • Since no-responses are clean by construction, we inject ambiguity explicitly (Table R3; see Q4).
AuthorsOpenReview ↗

Response to Reviewer q3G2 (2/2)

Q1. Both. Every logged anchor gets a propensity weight, whether its feedback is a positive response or a no-response, and the weights are self-normalized separately within each feedback type. Describing variant (D) as "positives only" was a labeling error; (D) uses the weight for both types, and the label is now corrected.


Q2. Fashion 18%, Clothing 12%, Grocery 13%, Health 9%, CDs 16%, and MovieLens 6% (App. A, Table 6).


Q3. The gain comes from the anchor entering the std, not from the std merely becoming non-zero. The anchor makes a zero std non-zero only when none of the 15 rollouts copies the logged item (m=0m=0), and that case carries at most 9% of the expected push toward alog⁡a^{\log} ([G1]); when some rollouts copy and others do not, the std is already non-zero and the anchor changes its size. Table R2 tests this directly: a jittered std without the anchor, the anchor in the mean only, and a 0.8× learning rate all stay within 0.3 pp of vanilla, while the anchor in both the mean and the std gives +2.3 / +2.2 pp (Health / CDs).


Q4. Yes. Since no-responses are clean in our base simulation, we ran three protocols that make them ambiguous on purpose (Table R3, Grocery): flipping a fraction ρ\rho of positives to no-responses, an examination model (each exposure is seen with probability 0.7), and latent positives (10% of distractors are items the user interacts with later). At ρ\rho = 30%, ABPO loses 1.9 pp of HR@5 versus 6.2 pp for the hard-negative rule, and the other two protocols keep the same ordering (27.1 vs. 22.4). The contribution of self-certainty also grows with ρ\rho, from +0.4 to +2.9 pp.


Q5. That is the vanilla row of Table R2 (16 sampled rollouts rewarded for matching alog⁡a^{\log}, no anchor): 16.1 / 30.9 HR@5 on Health / CDs, against 21.93 / 38.03 for ABPO. To check whether the extra group member is what helps, we also ran 16 rollouts plus the anchor (G=17G=17); it gives 18.5 / 33.2, the same as 15 rollouts plus the anchor (18.4 / 33.1). So the gain comes from the logged action being in the group, not from a larger group.


Q6. We use λsc=0.1\lambda_{\text{sc}}=0.1 (self-certainty weight), τ=1.0\tau=1.0 (propensity softmax temperature), δ=0.01\delta=0.01 (SNIPS shrinkage), and a weight clip of c=10c=10. Varying each around its default barely changes the results (Rec-R1, HR@5; default in bold):

  • λsc∈{0.05,0.1,0.2}\lambda_{\text{sc}}\in\{0.05, 0.1, 0.2\}, Health: 21.6 / 21.93 / 21.7
  • τ∈{0.5,1,2}\tau\in\{0.5, 1, 2\}, Grocery: 27.5 / 27.86 / 27.4
  • δ∈{0.001,0.01,0.1}\delta\in\{0.001, 0.01, 0.1\} and c∈{5,10,20}c\in\{5, 10, 20\}, Grocery: within 0.3 pp and 0.4 pp of the default

No sweep moves HR@5 by more than 0.5 pp.


Q7. ABPO was updated weekly on newly collected logs, so the three months are reporting windows (about four updates each), not update rounds. Table R5 gives user-clustered zz, pp, and 95% CIs for ABPO vs. ML-Retrain in each month and for the pooled period.

Reviewer_JwixOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work addresses the contextual-bandit feedback problem faced by generative LLM-based recommend systems when they are continuously updated after deployment: the logs only contain items exposed by the prior serving policy, leading to exposure bias, and the feedback consists only of positive responses and non-responses, with the latter being highly ambiguous. To address this issue, the authors propose ABPO, which inserts logged exposed items as anchors into the rollout group of GRPO and uses these anchors to calibrate the group-relative baseline; applies SNIPS to correct the off-policy contribution of the anchors; and adopts an asymmetric reward design for positive responses and non-responses, incorporating self-certainty as a reliability signal for non-response samples. The experiments span five Amazon domains and MovieLens, cover multiple LLM-Rec backbones, and report online A/B results.

Critical strengths
  1. This paper proposes the ABPO framework for continual LLM-Rec updates under policy-shaped contextual bandit feedback. The paper points out that deployment logs contain feedback only for items exposed by the prior policy, and that naively updating on such logs can lead to self-reinforcement of positive responses and over-penalization of no-responses. ABPO combines GRPO with exposure-bias correction and feedback-ambiguity handling, aiming to stably update generative recommend models under partial supervision and off-policy logs.

  2. The method design is sound. Using logged exposures as anchors rather than ground-truth labels to calibrate the GRPO baseline is a concise idea that aligns with the structure of bandit feedback. SNIPS is applied only to the anchor baseline and does not directly impose a PPO ratio on the anchors; this design is insightful, and the overall approach is relatively clear.

Critical weaknesses
  1. The reliability of the self-certainty reward lacks sufficient validation, as model confidence does not necessarily reflect recommendation correctness. The paper uses the model’s own output confidence as an auxiliary reward in no-response scenarios, but there is no supporting evidence for the relationship between model confidence and either the validity of recommendation results or user preferences.

  2. The theoretical analysis in this work is very weak. Although the authors provide a theoretical analysis of baseline calibration in the appendix, the current theory mainly demonstrates how anchors change the baseline; it does not prove that such changes can improve the expected return of the final policy or the regret results that matter most for bandit-based policies.

Questions for authors

Q1. Self-certainty measures the concentration of the model's output distribution, but high confidence does not necessarily imply recommendation correctness. When model confidence is inconsistent with real user preferences, does the self-certainty reward reinforce incorrect recommendations? Is this reward still effective across different user groups, long-tail items, and cross-domain recommendation scenarios?

Q2. Can it be proven that introducing anchors improves the long-term cumulative reward or regret?

3 comments
AuthorsOpenReview ↗

Response to Reviewer Jwix

Thank you for the careful reading; your description of where SNIPS enters (the anchor's contribution, no PPO ratio on the anchor) is exactly right. We address each point below.


W1 / Q1 (Self-Certainty). Addressed in Global Response [G5] and [G2].

  • SC never rewards a match; it only softens the penalty inside a no-response group, detached and group-relative.
  • When confidence and preference disagree the penalty still applies. AUROC 0.68–0.71 shows SC is a weak, tempering signal; under injected false negatives its gain rises (+0.4 → +2.9 pp) while a hard-negative rule degrades 3× faster.
  • Long-tail share 18.2% vs 16.8% (No-Update); by user-activity tercile the gain is 2.1 / 2.3 / 2.4 pp. Cross-domain transfer is not evaluated and not claimed.

W2 / Q2 (Theory, Regret). Addressed in Global Response [G1].

  • Proven: at the finite group size GRPO uses, the copy advantage is strictly decreasing in the anchor weight (§3.5 Remark), and m=0m=0 is bounded credit assignment. This is the quantity that decides whether an update amplifies past exposures.
  • Not proven, and not claimed: a regret or expected-return bound. The return-level evidence is empirical: best HR@5 / NDCG@5 in 24 of 24 backbone × domain cells while the other rules fall below No-Update in 63 of 72 HR@5 cells, plus the online gain.

We have incorporated the changes described in our rebuttal into the revised manuscript and highlighted them in red.

Thank you for the authors’ response. The additional analysis and experiments provided by the authors have addressed my main concerns, and I am willing to raise my score to 3.

Thank you very much for taking the time to consider our response and revisit your assessment. We sincerely appreciate your constructive feedback, which helped us strengthen the paper.

Comments on the paper

7 comments
AuthorsOpenReview ↗

Global Response (4/4)

[G6] Corrections and Revisions (all reviewers)


Takeaway. Re-checking the submission against the reviews, we found and fixed the inconsistencies below.


  • Online A/B Test (§4.5, App. D): tests were impression-level → user-clustered ([G3]).
  • SNIPS Scope: the "(D) positives only" label corrected to both types ([G4]).
  • Fig. 4 → numeric table (six domains).
  • Table 2: No-Update column added; metric and 0–100 scale defined in §4.3.
  • Theory: the mean-only Remark in §3.5 is replaced by a closed-form finite-sample damping Remark; no regret or expected-return claim.
  • Details: positive-feedback rates (App. A, Table 6); clip w≤10w\le 10 (Eq. 4); λsc=0.1\lambda_{\text{sc}}=0.1 (§3.4.1).
  • Presentation: matching rate defined before Fig. 2; SC described as detached and group-relative (§3.4.1).
  • Scope: SC measures concentration, not correctness (§3.4.1); online, we claim only the pooled gain over ML-Retrain (§4.5, App. D).
AuthorsOpenReview ↗

Global Response (3/4)

[G4] (New Baselines) Standard Off-Policy Baselines Added; the Weighting Modulates Exposure, It Does Not Make the Objective Unbiased. (q3G2 W2/Q1, apHm W1, J119 W2)


Takeaway. We adopt Reviewer apHm's distinction: the weight modulates how much the logged exposure counts, under the current policy and per feedback type; it does not make the objective unbiased. Against IPS and doubly-robust learning on the same logs, ABPO is best on all three domains tested.


  • Estimand. With w=eold/e0w=e_{\text{old}}/e_0, self-normalized within each stratum yy: E[wf∣y]E[w∣y]=Ea∼eold[f(a)∣y]\frac{\mathbb{E}[w f\mid y]}{\mathbb{E}[w\mid y]}=\mathbb E_{a\sim e_{\text{old}}}[f(a)\mid y] under response consistency and within-stratum overlap (§3.3); δ\delta is a shrinkage stabilizer. Renamed "SNIPS-style anchor-level exposure calibration".
  • Why not a full off-policy return. GRPO consumes a group-relative advantage, not a return; a propensity-weighted return brings back the single-sample variance the group baseline removes. The anchor acts inside that normalization, which pointwise IPS/DR do not touch.
  • OPL baselines (Table R6). IPS is high-variance here (max weight 13.7, ESS/∣B∣=0.61\mathrm{ESS}/|B|=0.61) because a generative policy's item propensities are peaked: it falls below No-Update on Grocery and CDs. DR trails ABPO by 1.7–4.3 pp. Stratifying the normalization by feedback type alone is worth +1.1 / +1.8 / +2.0 pp.
  • Diagnostics. y=1y=1 rate 6–18%; ESS/∣By∣\mathrm{ESS}/|B_y| 0.61–0.84; max ww 13.7, clipped at 10; clip ∈ {5, 10, 20} moves HR@5 ≤ 0.4 pp; PPO ϵ\epsilon ∈ {0.1, 0.2, 0.3} ≤ 0.6 pp.
  • SNIPS scope (q3G2 Q1). Both feedback types; the weight is normalized within its own stratum (Eq. 5)).

Table R6. Off-policy baselines (Rec-R1, HR@5)

Update ruleGroceryCDsHealth
No-Update25.9332.599.69
IPS policy gradient23.930.414.8
Doubly robust26.234.117.6
ABPO w/o stratified weighting26.836.219.9
ABPO27.8638.0321.93

[G5] (New Analysis) Self-Certainty Tempers the No-Response Penalty; It Is Not a Correctness Signal. (Jwix W1/Q1, apHm W4, q3G2 Q6)


Takeaway. Confidence is not correctness, and the design does not use it as such: self-certainty (SC) is a detached, group-relative term that only softens the penalty on no-response rollouts. It is a modest signal (AUROC 0.68–0.71 for held-out relevance), which is exactly why it tempers rather than replaces the penalty. Its gain grows with ambiguity, it beats four simpler rules, and it does not concentrate exposure.


  • What SC does. No-response group: rj=−1[aj=alog⁡]+rfmtj+λsc rscjr^j=-\mathbb{1}[a^j=a^{\log}]+r^j_{\text{fmt}}+\lambda_{\text{sc}}\,r^j_{\text{sc}}, with rscr_{\text{sc}} the mean token-wise KL(U ∥ πθold)\mathrm{KL}(U\,\|\,\pi_{\theta_{\text{old}}}), detached, λsc=0.1\lambda_{\text{sc}}=0.1. Advantages are group-relative, so only within-group differences matter; SC retains the explicit negative matching penalty and serves as an auxiliary signal, rather than replacing the observed feedback.
  • Confident-but-wrong? (Jwix Q1, apHm W4) (i) AUROC of SC for the held-out relevance of no-response items: 0.68 (Grocery) / 0.71 (Health). (ii) Under false negatives SC's gain rises while a hard-negative rule degrades 3× faster (Table R3). (iii) Long-tail share 18.2% vs 16.8% and coverage 11.4% vs 12.1% for No-Update, while every competing rule lowers both.
  • Simpler alternatives (Table R7). Ignore no-responses 20.3, tuned negative scaling 20.9, entropy reward 20.6, confidence-weighted logged item 20.7 vs SC 21.93 (Health HR@5).
  • Not the method alone. Over vanilla GRPO (Health / CDs, HR@5): SC alone +0.8 / +0.6 pp; anchor alone (C) +2.3 / +2.2; anchor + SC (E) +2.9 / +3.0; full ABPO (F) +5.8 / +7.1. The anchor is primary; SC complements it.
  • Hyperparameters (q3G2 Q6). Defaults λsc=0.1\lambda_{\text{sc}}=0.1, τ=1.0\tau=1.0, δ=0.01\delta=0.01, c=10c=10. Rec-R1 HR@5 around each default: λsc∈{0.05,0.1,0.2}\lambda_{\text{sc}}\in\{0.05,0.1,0.2\} on Health 21.6 / 21.93 / 21.7; τ∈{0.5,1,2}\tau\in\{0.5,1,2\} on Grocery 27.5 / 27.86 / 27.4; δ∈{0.001,0.01,0.1}\delta\in\{0.001,0.01,0.1\} within 0.3 pp. Every sweep moves HR@5 by ≤ 0.5 pp.
  • Scope. By user-activity tercile the SC gain is 2.1 / 2.3 / 2.4 pp (ρ\rho = 30%), not confined to heavy users. Cross-domain transfer is not evaluated and not claimed.

Table R7. Simpler tempering rules vs SC (Health, Rec-R1, HR@5)

VariantHR@5
Ignore no-responses entirely20.3
Tuned negative-reward scaling20.9
Entropy reward20.6
Confidence-weighted logged item20.7
Self-certainty (ours)21.93

Coverage and long-tail exposure (Grocery, Rec-R1); Table 2 Revised.

Coverage@5Long-tail share of top-5Div@5
No-Update12.1%16.8%27.64
DEALRec7.8%10.2%27.57
RL-Rec8.7%13.5%27.71
UL-Rec8.1%11.4%26.83
ABPO11.4%18.2%28.52
AuthorsOpenReview ↗

Global Response (2/4)

[G2] (New Experiments) Exposure Bias Is a Selection Effect in Our Logs; Ambiguity Is Now Injected Explicitly. (q3G2 W3/Q2/Q4, apHm W2, Jwix Q1)


Takeaway. Agreed: the base simulation has exact labels and clean no-clicks. What it does contain is exposure bias as selection: a positive is observed only where the logging policy exposed the true item, and that selection alone costs −13% / −20% HR@5 (Fig. 1). Choosing the same number of positives at random cuts the loss to −2% / −3%, and on the same biased logs ABPO gains 14% / 23% where a hard-negative update loses 32% / 53% (Table R4).


  • What the logs contain. y=1y=1 iff the logged item is the true next item (Fashion 18%, Clothing 12%, Grocery 13%, Health 9%, CDs 16%, and MovieLens 6%). Labels are exact, but which contexts get a positive is decided by e0e_0: missing-not-at-random selection, i.e., exposure bias in bandit-log form.
  • Fig. 2 is a training-log matching rate, not accuracy: the share of rollouts that reproduce the logged item. Vanilla's climbs to ~60% while its held-out HR@5 stays below No-Update on CDs (30.9 vs 32.59): it memorizes exposures.
  • Ambiguity injected (Table R3). False-negative flips (ρ\rho of y=1y=1 relabeled 0), an examination model (examined w.p. 0.7), and latent positives (10% of distractors are the user's future items). The SC contribution rises +0.4 → +2.9 pp with ρ\rho; ABPO degrades 3× slower than a hard-negative rule.
  • Real exposure data is the online test (§4.5); §4.1 now states the simulation's scope.
  • Revised: §4.1 states the simulation's scope; positive-feedback rates in App. A (Table 6); the matching rate is defined in §2.1 and the Fig. 2 caption.

Table R3. Injected ambiguity (Grocery, Rec-R1, HR@5)

cleanρ\rho = 10%ρ\rho = 30%ρ\rho = 50%
SC contribution (E − C)+0.4+1.1+2.3+2.9
ABPO, change vs clean–−0.6−1.9−3.1
Hard-negative rule, change vs clean–−2.4−6.2−9.7

Examination model and latent positives keep the ordering: ABPO 27.1 vs hard-negative 22.4.

Table R4. Selection vs. update rule (Fig. 1 setting, Rec-SAVER; change vs No-Update)

Update dataRuleGrocery HR@5MovieLens HR@5Div@5 (Grocery / MovieLens)
Positives selected by e0e_0plain (+1 on match)−13%−20%−41% / −48%
Same number of positives, at randomplain (+1 on match)−2%−3%−6% / −8%
All e0e_0 logs (positives + no-responses)hard-negative (±1 on match)−32%−53%−23% / −28%
All e0e_0 logs (positives + no-responses)ABPO+14%+23%0% / −5%

Rows 1–2 differ only in which contexts receive a positive; rows 3–4 only in the update rule.

[G3] (Re-analysis) Online A/B: Unit Corrected, User-Clustered Tests, Power, and a Narrower Claim. (q3G2 W2/Q7, apHm W3, J119 W1)


Takeaway. Users were randomized, but the App. D tests used impressions, which overstates zz. With user-clustered SEs, the 3-month gain over the concurrent ML-Retrain arm remains significant (+0.34 pp, z=3.83z=3.83); February alone does not (z=0.68z=0.68). We now claim only this pooled advantage: neither our monthly CTR rise nor the growing monthly gap is claimed as evidence.


  • Clustered re-analysis (Table R5). 8.4 impressions per user, click ICC 0.18 → clustered zz = impression z/3.1z/\sqrt{3.1}, bootstrap-confirmed. Against Popularity and GRPO-Update every month stays at z≥23z\ge 23.
  • Power (J119). Minimum detectable effect at 80% power: ≈ 0.41 pp per month, ≈ 0.25 pp pooled over 3 months. Single months are underpowered for a +0.1–0.4 pp effect, so the pooled +0.34 pp is the primary estimate.
  • Unequal traffic (q3G2). Impression shares: Popularity 14%, ML-Retrain 18%, GRPO-Update 25%, ABPO 43%, constant across months (Table 8); verified LLM-Rec variants received larger shares by production policy (§4.5). Users were hash-randomized, so unequal shares affect power, not bias; a traffic-matched subsample gives 7.41% vs 7.10% (p=0.003p=0.003, clustered).
  • Trend. Own-arm Feb→Apr z=3.28z=3.28, Mar→Apr n.s.; difference-in-differences vs ML-Retrain ≈ 2.0, borderline. The pooled randomized comparison supports a CTR advantage over ML-Retrain. Monthly estimates increase numerically, but we do not claim a statistically established increase in the treatment effect over time (apHm).
  • Design. 3 months (Feb–Apr); the policy was updated weekly (months are reporting windows, not update points; q3G2 Q7); clustered 95% CIs per month.

Table R5. ABPO vs. ML-Retrain, user-clustered

Δ CTRzz (impression)zz (clustered)pp (clustered)Clustered 95% CI
Feb+0.10 pp1.200.680.49[−0.19, +0.39] pp
Mar+0.40 pp4.672.650.008[+0.10, +0.70] pp
Apr+0.53 pp5.883.34< 0.001[+0.22, +0.84] pp
3-month (pooled)+0.34 pp6.743.83< 0.001[+0.17, +0.51] pp

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.