- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies whether popularity-debiasing judgments from a LLM can be transferred to a smaller recommendation model. It introduces counterfactual sensitivity probes, constructed by pairing real items with overlapping attributes but different popularity levels. Results show that student learns preferences without inheriting explicit popularity.
Critical strengths
-
The idea of constructing counterfactual pairs is interesting, though hard to guarantee and not that scalable. It provides a potentially useful setting for investigating whether preference models rely excessively on popularity.
-
The paper honestly reports cases where the proposed approach underperforms its baselines.
Critical weaknesses
-
The definition and construction of counterfactual probes are not convincing enough. The sensitivity pairs consist of different real items matched on several attributes, while in principle, the
ideal pairscan be constructed with interventions that vary popularity only while holding other factors fixed. In practice, it is also quite difficult to obtain such kind of counterfactual pairs. -
The method novelty is limited. Counterfactual pairs, curriculum training and teacher-generated preference labels are established techniques.
-
The 'sensitivity accuracy score' is measured by comparing model answers to labels made by a LLM teacher. This teacher was specifically instructed to use popularity as a tie‑breaker when choosing between options. Then, the paper shows that the teacher picks the more popular option in about 73% of these test cases, which closely matches the 72.91% accuracy you’d get by simply always guessing the more popular choice. So, a high sensitivity accuracy score might just mean the model agrees with the teacher’s habit of favoring popular answers.
Questions for authors
-
The teacher uses popularity as a tie-breaker and the popularity-only heuristic outperforms all trained methods on sensitivity accuracy, how can this metric distinguish genuine popularity debiasing from teacher imitation?
-
How does the proposed curriculum compare with SFT and DPO trained on the same teacher-labeled pairs? Would a simple rule based on historical attribute overlap provide similar sensitivity accuracy?
-
The IPS baseline implementation details. The IPS baseline approximates exposure propensity using item popularity percentile. How sensitive are the results to this approximation?
Thank you for the direct questions; they pointed at exactly the experiments we needed to run. All numbers are new, on the same eval pairs (setup in the global response; full tables in the revised PDF). Where the data undercuts our own methods, we say so.
Q1 (and W3): how can the metric distinguish genuine popularity debiasing from teacher imitation? We agree the paper's metric alone could not, and built the instruments that can:
Q2: curriculum vs SFT and DPO on the same teacher-labeled pairs? Would a simple attribute-overlap rule provide similar sensitivity accuracy? All three now run, and the answer restructures the paper:
Q3: how sensitive are results to approximating exposure propensity by popularity percentile? Re-running the IPS baseline (same seed, same protocol): η=0.001 → 70.13, η=0.05 (code's value) → 64.43, η=0.1 → 71.81, w_max=5 → 66.11. All variants are entropy-stable; the spread (64–72) is within the baseline's own seed-noise band (60.4–67.5 across 3 seeds). This question also surfaced an inconsistency we will fix: the appendix states η=0.01 but the code uses η=0.05 (the main text's "≈1/η=20×" matches the code); the revision corrects the appendix.
W1 (counterfactual construction falls short of an intervention). Agreed, and the revision no longer lets the word claim more than it delivers: the probes are popularity-contrastive minimal pairs over real items — a diagnostic and training signal that isolates popularity as nearly as logged data affords, explicitly not a causal intervention holding all else fixed (that would require randomized exposure, unavailable in any historical log). We keep real-item pairing to avoid text-perturbation artifacts; the price is the residual confounds you identify, which we now measure rather than only acknowledge (cardinality-controlled subset: gaps widen, so conclusions do not rest on the confound).
W2 (methodological novelty limited). Fair on the components — GRPO, curriculum ordering, and teacher preference labeling are established, and the new SFT result shows the established simpler method wins here. The contribution we ask to be evaluated on is the probe construction as a computable diagnostic: it turned "does the model track content or popularity" into measurable quantities with computable reference points (the two rules, the popularity heuristic, the teacher itself), and it is what surfaced every finding above — including several unflattering to our own methods. The training pitfalls (now with more instances across seeds) and the measured cost analysis are instrumentation, not algorithmic novelty. The contributions list in the revision says exactly this and nothing stronger.