acceptodds
Under review as a conference paper at ICLR 2027

Risk-Controlled Personalized Popularity Alignment for Frozen Collaborative Filtering

Abstract

Popularity correction can improve agreement with user histories while reducing ranking utility. We study when to apply such edits to frozen recommenders under explicit utility-loss budgets. Our interface separates candidate construction, user ordering and risk authorization. Given the resulting candidate lists, independent calibration selects the largest score prefix whose positive-part harms for Recall and NDCG satisfy their respective budgets. Rejected edits return the original list exactly. Conformal risk control bounds marginal expected harm under exchangeability. We instantiate the interface with RCPA-CF, our risk-controlled popularity-alignment method, and compare CP and TTEN-style corrections. Evaluation covers three datasets, four backbones and five seeds, with disjoint calibration and evaluation users. Under the conservative RCPA setting, learned ordering reduces mean popularity calibration error (PCE) by 1.839%, versus 1.654% for gain-only and 1.626% for regret-only ordering. All three orderings meet both realized budgets in 56/60 runs; this is a batch diagnostic rather than a probability guarantee. Ordering gains concentrate in Gowalla BPR-MF and LightGCN. Historical calibration-pool replays show an additional benefit over a change-based heuristic in selected settings. Further analyses relate candidate choice and alignment metrics to retained benefit and user-level loss distributions. The interface thus enables selective popularity correction under explicit utility-loss budgets.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
recommender systems, popularity bias, collaborative filtering, popularity alignment, conformal risk control, risk-aware recommendation, post-hoc correction

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_1wUsOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RCPA-CF, a risk-controlled popularity-alignment framework for frozen collaborative filtering models. It aims to improve popularity alignment while limiting ranking utility loss. The method constructs popularity-corrected candidate lists, orders users based on predicted harm and alignment gain, and uses conformal risk control to authorize edits under separate Recall and NDCG budgets. Experiments on three datasets, four backbones, and five seeds show that learned ordering reduces mean popularity calibration error (PCE).

Critical strengths
  1. The framework separates candidate construction, user ordering, and risk authorization, allowing different popularity corrections to be applied under explicit utility-loss budgets.
  2. The experiments cover three datasets, four backbones, and five seeds, with comparisons against existing popularity correction methods and different ordering strategies.
Critical weaknesses
  1. Insufficient justification for the reliability of harm prediction. The proposed method predicts Recall and NDCG harm using 14 features derived from user histories, popularity statistics, frozen scores, and recommendation list changes. However, I am not fully convinced that these features can reliably predict the utility loss caused by popularity correction. Popularity alignment measures how well the recommendation list matches the user's historical popularity distribution, while NDCG depends on the relevance of the recommended items. Two users with similar historical statistics and recommendation changes may have very different NDCG losses after correction. Although the model can learn the expected harm from historical data, the paper does not clearly explain why these features are sufficient for reliable prediction. Moreover, Appendix Table 30 shows that harm-occurrence AUROC is close to or below 0.5 in several settings. I believe more evidence is needed to support the reliability of this prediction module, especially for predicting the magnitude of utility loss.
  2. Limited theoretical support for the learned harm prediction and ordering. The paper provides theoretical guarantees for controlling expected utility harm through conformal risk control. This is a useful property of the framework. However, my concern is that these guarantees do not directly establish the effectiveness of the learned harm predictor. The risk guarantee holds for any fixed ordering score under the stated assumptions, even when the harm prediction is inaccurate. Therefore, it remains unclear what theoretical benefit the learned predictor provides over simpler ordering strategies. Since the main motivation is to retain more popularity-alignment benefit under utility-loss budgets, I believe the theoretical connection between harm prediction accuracy and the final alignment benefit needs further clarification.
Questions for authors
  1. How can the authors justify that the proposed 14 features can reliably predict Recall and NDCG harm after popularity correction, especially when users with similar features may have different relevance outcomes?
  2. How does inaccurate harm prediction affect the effectiveness of learned ordering, and does the theoretical analysis provide any guarantee on the retained alignment benefit?
Reviewer_pVFhOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

Paper asks when to apply popularity corrections to a frozen recommender such that the recommendation utility loss is limited. It constructs an edited list, orders users by estimated harm relative to alignment gain, and uses conformal risk control to select which users receive the edit. Other users retain original recommendations. Experiments cover three datasets and four backbones.

Critical strengths

a) The positive-part loss formulation is useful as utility improvements for some users cannot cancel losses suffered by others.

b) The evaluation includes multiple backbones, and the findings are appropriately scoped.

Critical weaknesses

a) The presentation leaves considerable room for improvement and is difficult to understand in its current form. The introduction assumes terminology that it has not explained, and some methodological wording needs clarification. For exp. CP and TTEN appear in the abstract and introduction without expansion or a brief explanation at first use. A similar trend is visible throughout the paper. The introduction also blurs the distinction between popularity calibration and risk calibration. Page 5 states that the wrapper adds no trainable parameters, although Page 19 mentions fitted ridge coefficients. The appropriate clarification is that the backbone requires no retraining, while the ordering model is trained.

b) The methodological contribution beyond applying Conformal Risk Control (CRC) is unclear. The introduction presents the separation of candidate construction, ordering, and calibration as a central contribution. However, fixing the prediction procedure before independent risk calibration is already part of the standard CRC setup (Angelopoulos et al., 2024). The paper itself describes its risk guarantee as a specialization of multi-risk CRC (3.4). Although candidate construction and learned ordering are additional choices, the exact technical gap or research advancement beyond this application needs to be described.

c) I believe that, since the paper is about risk control, other conformal and risk-control methods should be compared and considered (e.g., LTT, RCPS, etc.).

d) PCE seems unsuitable as the main measure of popularity alignment. As the authors’ own counterexample shows, PCE can decrease even when recommendations deviate from an exact match to the historical popularity distribution. Why is it used as the main benefit objective?

Questions for authors

Please address the weaknesses.

Reviewer_WjWPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents a method for selective personalized popularity calibration of a frozen recommender system with explicit recall and normalized discounted cumulative gain (NDCG) loss constraints. The approach consists of three steps. First, within the top M candidate pool of the frozen model, it adjusts the scores of candidates based on the user's past popularity preferences to generate a candidate list (RCPA-CF). Second, it sorts users according to the ratio of the user's predicted harm to their PCE gain. Third, using an independent calibration set, it applies conformal risk control to identify the largest number of users who can be calibrated while meeting both budgets. Users outside of the identified set will retain their original base list. Theoretically, if the exchangeability assumption holds, then the expected marginal harm will not exceed the budget. The experiments include three datasets, four frozen backbones, and five seeds. The paper compares the approach to other candidate correction methods (e.g., CP and TTEN) and several simple sorting strategies.

Critical strengths

This paper has a clearly defined problem statement, as well as an elegant framework for addressing this problem. Specifically, the framework separates out the "how" of constructing a correction from the "who" that correction should be applied to. Additionally, the framework takes the average of one-sided harm to users; thus, a gain to one user cannot compensate for a loss experienced by another user. Unauthorized users will receive their original recommendations. The framework does not require modification of the recommendation system, nor does it require any additional training of parameters. Finally, deploying this framework is relatively inexpensive (approximately 0.045 milliseconds per user), making it suitable for implementation into existing systems.

There is an extremely strict experimental protocol and transparent reporting. The calibration and evaluation users are distinct. All settings, ordering rules and budgets are established prior to reading the calibration labels. The paper thoroughly documents each time a budget is exceeded, when there is no benefit from a particular setting, and when a baseline variant performs better than the proposed method. Additionally, it offers supplementary material to enable auditing. This enables the extent of the conclusions to be properly evaluated.

Critical weaknesses

The paper is hard to follow because the main argument is frequently broken up by qualifications and key details (tie handling, limitations of the PCE metric, ordering-stability requirements) are only in the appendices. The gains are also small: the average PCE reduction is modest and RCPA is clearly worse than CP and TTEN, which improve PCE much better under the same budget.

Questions for authors
  1. Why RCPA is the most feasible option, considering that CP and TTEN get much more PCE improvements with the same budget?
  2. What happens in a temporal split or with new users where exchangeability does not hold?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.