- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Framework for updating LLM-based recommenders using post-deployment feedback + anchoring updates to historically exposed items to correct for exposure bias. It treats positive and no-response feedback with self-normalized inverse propensity scoring. It treats positive feedback (clicks) differently from no-response feedback (no clicks), using the model's confidence to avoid over-penalizing unclear signals.
This framework showed good results on online and offline experiments.
Critical strengths
The authors spent quite some time working on the validation (offline and online), and they added several baselines + datasets. This helps build trust on their results and method.
Critical weaknesses
Their statistical analysis could use some more rigour. In particular, AB tests often require power analysis to ensure robustness of the statistical test - but there was no mention on how any approach might impact (positively or negatively).
In bandits work, propensity weighting to correct for selection biases is also quite common. They do make some comparisons with other correction biases, but it would be valuable to also have a more close comparison with a very well-known standard methods.
Questions for authors
The authors mentioned 'diversity' - but it is unclear its overall role and impact. Can't this be something learned? Or tuned? It looks like one of the goals is to have more 'diversity' (not recommend same item over and over), but one might be able to measure and include that on framework?
Considering how strong the 'no update' baseline was, and that naive contextual bandit updates might harm performance. Even if I add the proposed approach, the gains are very marginal at the cost of a lot of complexity to maintain the framework. Is there any additional motivation to an user to adopt this framework? Does it converge faster? Require less samples to obtain enough statistical power?
3 comments
Thanks for all the comments. Based on the authors feedback + global answer, I'll update my score from 2 to 3.
Thank you so much for your kind follow-up and for updating your score! We really appreciate your time and valuable feedback.
Response to Reviewer J119
Thank you for pressing on rigor and on practical value. We have revised the manuscript to address the reviewers' comments and highlighted the changes in red. We address each point below.
W1 (power). Addressed in Global Response [G3].
W2 (standard propensity baselines). Addressed in Global Response [G4].
Q1. (Clarification) Diversity Is a Reported Safety Outcome, Not a Tuned Objective.
Takeaway. Updating on bandit logs tends to concentrate exposure: positive-only and hard-negative updates cut diversity by 23–48% (Fig. 1), while ABPO stays close to the starting policy (0% / −5% on Grocery / MovieLens; Table R4). We therefore report diversity as a check that an update has not narrowed what users see, rather than as something we optimize.
Q2. (Clarification) The Gain Is a Safe Update, Not a Marginal One.
Takeaway. The comparison that matters in deployment is updating versus not updating. On the same logs, the other update rules fall below No-Update in 63 of 72 HR@5 cells, so a naive update often makes the deployed model worse. ABPO is best in all 24 backbone–domain settings on HR@5 and NDCG@5 (17 of 24 on HR@1), with a median HR@5 gain of +15.3% over No-Update. The gain is smallest on the strongest backbone, G²RPO (median +3.2%), and online it is +0.34 pp CTR, a 4.7% relative improvement.