- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes RCPA-CF, a risk-controlled popularity-alignment framework for frozen collaborative filtering models. It aims to improve popularity alignment while limiting ranking utility loss. The method constructs popularity-corrected candidate lists, orders users based on predicted harm and alignment gain, and uses conformal risk control to authorize edits under separate Recall and NDCG budgets. Experiments on three datasets, four backbones, and five seeds show that learned ordering reduces mean popularity calibration error (PCE).
Critical strengths
- The framework separates candidate construction, user ordering, and risk authorization, allowing different popularity corrections to be applied under explicit utility-loss budgets.
- The experiments cover three datasets, four backbones, and five seeds, with comparisons against existing popularity correction methods and different ordering strategies.
Critical weaknesses
- Insufficient justification for the reliability of harm prediction. The proposed method predicts Recall and NDCG harm using 14 features derived from user histories, popularity statistics, frozen scores, and recommendation list changes. However, I am not fully convinced that these features can reliably predict the utility loss caused by popularity correction. Popularity alignment measures how well the recommendation list matches the user's historical popularity distribution, while NDCG depends on the relevance of the recommended items. Two users with similar historical statistics and recommendation changes may have very different NDCG losses after correction. Although the model can learn the expected harm from historical data, the paper does not clearly explain why these features are sufficient for reliable prediction. Moreover, Appendix Table 30 shows that harm-occurrence AUROC is close to or below 0.5 in several settings. I believe more evidence is needed to support the reliability of this prediction module, especially for predicting the magnitude of utility loss.
- Limited theoretical support for the learned harm prediction and ordering. The paper provides theoretical guarantees for controlling expected utility harm through conformal risk control. This is a useful property of the framework. However, my concern is that these guarantees do not directly establish the effectiveness of the learned harm predictor. The risk guarantee holds for any fixed ordering score under the stated assumptions, even when the harm prediction is inaccurate. Therefore, it remains unclear what theoretical benefit the learned predictor provides over simpler ordering strategies. Since the main motivation is to retain more popularity-alignment benefit under utility-loss budgets, I believe the theoretical connection between harm prediction accuracy and the final alignment benefit needs further clarification.
Questions for authors
- How can the authors justify that the proposed 14 features can reliably predict Recall and NDCG harm after popularity correction, especially when users with similar features may have different relevance outcomes?
- How does inaccurate harm prediction affect the effectiveness of learned ordering, and does the theoretical analysis provide any guarantee on the retained alignment benefit?