acceptodds
Under review as a conference paper at ICLR 2027

Counterfactual Sensitivity Probes: Distilling Popularity-Robust Judgment into Fast Sequential Recommenders via GRPO

Abstract

Sequential recommenders trained on implicit feedback inherit popularity bias, and while LLMs can reason explicitly about this confound, such reasoning is too slow for production serving — a claim we verify by direct measurement rather than assumption. We introduce counterfactual sensitivity probes — real item pairs matched on content but deliberately mismatched on popularity — as both a training signal and a diagnostic for whether a model's preference tracks content or popularity, and use them to distill a large teacher's popularity-debiasing judgment into a small, non-reasoning student via two GRPO recipes: single-stage training on the full teacher-labeled mix, and a two-stage curriculum that warms up on the diagnostic pairs first. Along the way, we fix a previously undiagnosed unbounded policy-entropy growth in GRPO training via an adaptive KL-to-reference controller and a capped IPS reward, enabling one pre-registered stopping rule applied uniformly to both recipes and two classical baselines (alignment-only, IPS-weighted), and we pair the resulting accuracy benchmark with a deployment-oriented cost analysis, measured rather than assumed: the distilled student is faster and higher-throughput than the teacher, while curriculum's two-stage training costs a single-stage run, driven by job-launch overhead rather than extra computation. On Amazon Beauty, results are honestly mixed: curriculum is the only method that clears both baselines (68.79% sensitivity accuracy).

Primary area:
reinforcement learning
Keywords:
sequential recommendation, popularity debiasing, GRPO distillation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_CRQTOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies whether popularity-debiasing judgments from a LLM can be transferred to a smaller recommendation model. It introduces counterfactual sensitivity probes, constructed by pairing real items with overlapping attributes but different popularity levels. Results show that student learns preferences without inheriting explicit popularity.

Critical strengths
  1. The idea of constructing counterfactual pairs is interesting, though hard to guarantee and not that scalable. It provides a potentially useful setting for investigating whether preference models rely excessively on popularity.

  2. The paper honestly reports cases where the proposed approach underperforms its baselines.

Critical weaknesses
  1. The definition and construction of counterfactual probes are not convincing enough. The sensitivity pairs consist of different real items matched on several attributes, while in principle, the ideal pairs can be constructed with interventions that vary popularity only while holding other factors fixed. In practice, it is also quite difficult to obtain such kind of counterfactual pairs.

  2. The method novelty is limited. Counterfactual pairs, curriculum training and teacher-generated preference labels are established techniques.

  3. The 'sensitivity accuracy score' is measured by comparing model answers to labels made by a LLM teacher. This teacher was specifically instructed to use popularity as a tie‑breaker when choosing between options. Then, the paper shows that the teacher picks the more popular option in about 73% of these test cases, which closely matches the 72.91% accuracy you’d get by simply always guessing the more popular choice. So, a high sensitivity accuracy score might just mean the model agrees with the teacher’s habit of favoring popular answers.

Questions for authors
  1. The teacher uses popularity as a tie-breaker and the popularity-only heuristic outperforms all trained methods on sensitivity accuracy, how can this metric distinguish genuine popularity debiasing from teacher imitation?

  2. How does the proposed curriculum compare with SFT and DPO trained on the same teacher-labeled pairs? Would a simple rule based on historical attribute overlap provide similar sensitivity accuracy?

  3. The IPS baseline implementation details. The IPS baseline approximates exposure propensity using item popularity percentile. How sensitive are the results to this approximation?

1 comment

Thank you for the direct questions; they pointed at exactly the experiments we needed to run. All numbers are new, on the same eval pairs (setup in the global response; full tables in the revised PDF). Where the data undercuts our own methods, we say so.

Q1 (and W3): how can the metric distinguish genuine popularity debiasing from teacher imitation? We agree the paper's metric alone could not, and built the instruments that can:

  • Popularity-pick rate (teacher-independent): students pick the more popular candidate 50–65% of the time; the teacher itself picks it 70.8%. Imitating the teacher's popularity habit would put this at ~70.8%; it is not. Agreement is earned from both label directions (T− 60.9–94.3%, T+ 56.9–89.6%) — students reproduce the content-comparison component, not the tie-breaker.
  • True-next-item rate (no teacher label): SFT 74.50, curriculum 62.42, align-only 59.06, IPS 58.25, single-stage 55.56, chance 50%.
  • The imitation hypothesis, tested against the teacher itself: a rule holding the same statistics the teacher sees, with the teacher's own tie-break instruction, agrees with the teacher on 92.28% of pairs — and where the teacher deviates from that formula (7.7% of pairs), its deviations track the actual next item worse than the formula (34.8% vs 65.2%). The precise statement: the teacher mostly runs a formula, the students encode a partial, attribute-only version of it (below the statistics-free rule's 78.86%), and nobody is imitating popularity.

Q2: curriculum vs SFT and DPO on the same teacher-labeled pairs? Would a simple attribute-overlap rule provide similar sensitivity accuracy? All three now run, and the answer restructures the paper:

  • SFT: 90.94% teacher-agreement / 74.50% true-next / 88.9% swap consistency — statistically indistinguishable from the teacher's own formula (92.28%, McNemar p=0.65) and dominating every GRPO recipe (best GRPO: 68.79; p<10⁻⁴). DPO: 48.99% — fails (its implicit reward cannot separate single-token completions; training loss stays at log 2).
  • Attribute-overlap rule from the student's own input (no LLM, no statistics): 78.86% / 62.42% — not merely "similar", it exceeds every GRPO recipe and ties curriculum exactly on true-next (p=1.0). The information gap is not binding: the student's input supports ~90% agreement (SFT proves it), and our GRPO training extracted far less of it. We report this as the study's central negative finding; the revised paper re-scopes accordingly — the residual case for online RL over SFT is extensibility to non-static rewards (learned reward models, interactive objectives), a hypothesis, not a demonstrated advantage.

Q3: how sensitive are results to approximating exposure propensity by popularity percentile? Re-running the IPS baseline (same seed, same protocol): η=0.001 → 70.13, η=0.05 (code's value) → 64.43, η=0.1 → 71.81, w_max=5 → 66.11. All variants are entropy-stable; the spread (64–72) is within the baseline's own seed-noise band (60.4–67.5 across 3 seeds). This question also surfaced an inconsistency we will fix: the appendix states η=0.01 but the code uses η=0.05 (the main text's "≈1/η=20×" matches the code); the revision corrects the appendix.

W1 (counterfactual construction falls short of an intervention). Agreed, and the revision no longer lets the word claim more than it delivers: the probes are popularity-contrastive minimal pairs over real items — a diagnostic and training signal that isolates popularity as nearly as logged data affords, explicitly not a causal intervention holding all else fixed (that would require randomized exposure, unavailable in any historical log). We keep real-item pairing to avoid text-perturbation artifacts; the price is the residual confounds you identify, which we now measure rather than only acknowledge (cardinality-controlled subset: gaps widen, so conclusions do not rest on the confound).

W2 (methodological novelty limited). Fair on the components — GRPO, curriculum ordering, and teacher preference labeling are established, and the new SFT result shows the established simpler method wins here. The contribution we ask to be evaluated on is the probe construction as a computable diagnostic: it turned "does the model track content or popularity" into measurable quantities with computable reference points (the two rules, the popularity heuristic, the teacher itself), and it is what surfaced every finding above — including several unflattering to our own methods. The training pitfalls (now with more instances across seeds) and the measured cost analysis are instrumentation, not algorithmic novelty. The contributions list in the revision says exactly this and nothing stronger.

Reviewer_uvPqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies how to transfer recommendation judgments that reduce popularity bias from a large model to a smaller one. The authors construct item pairs with similar attributes but substantially different popularity, use a Qwen3-32B teacher to generate preference labels, and train a Qwen3-1.7B student via GRPO. The paper compares direct training on mixed data with a two-stage curriculum that first trains on these item pairs. Experiments on Amazon Beauty show that the curriculum approach outperforms the two trained baselines on sensitivity accuracy but still falls short of a simple popularity-only rule on that metric; several methods achieve higher alignment accuracy. The paper also analyzes training problems and reports training and inference costs.

Critical strengths
  1. The authors pair real items with similar attributes but substantially different popularity for training and evaluation. This approach requires no artificial changes to item text and offers a useful way to examine whether recommendation models rely too heavily on popularity, especially when item information is limited.
  2. The paper openly reports results that fall below a simple baseline and discusses problems such as position bias, training instability, and checkpoint selection. It also reports measured training and inference costs, providing useful practical information for future research.
Critical weaknesses
  1. The evaluation does not establish that popularity bias is reduced. The teacher selects the more popular item in approximately 73% of sensitivity pairs, and the popularity-only heuristic achieves 72.91% sensitivity accuracy, higher than every trained method (best: curriculum at 68.79%). The current metric therefore largely measures agreement with the teacher’s popularity tie-breaking, and the results cannot distinguish reduced reliance on popularity from a failure to learn the teacher’s judgments. The evidence is further weakened by a single dataset, no full ranking results, a single seed, and near-chance swap consistency (44.75% for single-stage and 52.93% for curriculum).

  2. Key baselines are missing, and the counterfactual probes are confounded. The paper does not compare against SFT, DPO, or simple attribute-matching rules; since the training objective is to match binary teacher labels, it remains unclear whether GRPO and a large teacher provide benefits over these simpler alternatives. In addition, the counterfactual pairs share only some attributes and may differ in attribute count and quality, as acknowledged in Section 3.3. The paper does not assess how much these differences affect the teacher’s labels, leaving the validity of these pairs as a debiasing training signal unclear.

Questions for authors
  1. On samples where the teacher prefers the less popular item, does your method outperform the baselines?
  2. Beyond agreement with the teacher, what evidence indicates that the model reduces popularity bias?
  3. Why does choice consistency fall to only 52.93% after the candidate order is swapped?
  4. What advantages does GRPO provide over SFT, DPO, and simple attribute-matching rules?
  5. How do you rule out the influence of differences in attribute count, content, and quality between counterfactual item pairs?
2 comments

Thank you for the sharp questions — each is answered with a new measurement on the same eval pairs (setup and full tables: global response and revised PDF). Where the data undercuts our own methods, we say so.

Q1 (samples where the teacher prefers the less popular item). Accuracy on T− (n=87) and T+ (n=211):

Align-onlyIPSSingleCurriculumSFT
T−60.9271.2663.2267.8294.25
T+64.4561.6156.8769.1989.57

IPS wins T− and loses T+ (the most anti-popularity method); curriculum is the most balanced; SFT wins both. Caveat: the teacher's own T− picks match the actual next item only 40.2% of the time (below coin) — the T− labels are the least behavior-validated part of the metric, so no claim rests on T− alone.

Q2 (evidence beyond agreement with the teacher). Three teacher-independent measurements:

  1. True-next-item rate (the positive is one of the two candidates by construction): SFT 74.50 (CI 69.3–79.1) ≈ teacher itself (73.49) > curriculum 62.42 > align-only 59.06 > IPS 58.25 > single 55.56 (chance 50%; DPO 48.99 fails).
  2. Popularity-pick rate: students pick the more popular candidate 50–65% of the time vs the teacher's 70.8% and the heuristic's 100%.
  3. Swap decomposition (Q3): inconsistency is input symmetry, not popularity dependence. Counterweight (quantified): popularity is genuinely predictive here — the actual next item is the more-popular twin 79.19% of the time (positives sampled by interaction frequency), so even SFT does not beat the popularity heuristic on raw next-item prediction. This is the benign-bias confound we cited; the revision re-scopes claims: on this log, reduced popularity reliance cannot mean beating the heuristic at replaying the log.

Q3 (why choice consistency falls to 52.93% after the swap). Sensitivity twins are near-textually-identical, so the swapped prompt is nearly the same string — little but position remains to break the symmetry. Evidence: (i) every method's swap consistency is 15–25 points lower on sensitivity than alignment pairs (align-only 72.9 vs 53.7); (ii) on the 13 pairs with literally identical attribute sets the two prompts are the same string and swap consistency is exactly 0.0 for all models including SFT; (iii) it tracks input symmetry, not training health: single-stage is least consistent (32.7), IPS (most stable) most consistent (60.7), SFT 88.9. The decomposition is added to the revision.

Q4 (advantages of GRPO over SFT, DPO, attribute-matching rules). All three now run on the identical data:

  • SFT: 90.94% agreement / 74.50% true-next / 88.9% swap consistency — dominates every GRPO recipe on every metric (vs curriculum: p<10⁻⁴ agreement, p=2.6×10⁻⁴ true-next, exact McNemar). DPO: 48.99% (at chance; single-token completions give its implicit reward no discrimination — training loss stays at log 2).
  • Attribute-overlap rule (no LLM, computable from the student's own input): 78.86% / 62.42%; with the teacher's popularity tie-break: 92.28% / 75.84%.
  • Plain answer: on this benchmark GRPO provides no measured advantage over SFT and sits below both rules on agreement. Our Appendix's defense of GRPO as a methodological commitment is answered negatively by data for binary-label distillation; the residual case for online RL is extensibility to non-static rewards (learned reward models, interactive objectives) — a hypothesis, not a demonstrated advantage.

Q5 (ruling out attribute count / content / quality differences). Cardinality is controlled: on the 136 equal-tag-count pairs, curriculum 70.59 > align-only 62.50 > IPS 59.56 > single 55.15 (SFT 86.03) — gaps widen on the clean subset, so the comparisons do not rest on the confound (itself quantified: the teacher picks the fewer-tag item on 72.8% of unequal pairs). Content differences are the probe design (match content, vary popularity). Quality cannot be controlled in this dataset (no ratings/review signals) — stated as a limitation.

W1 (evaluation does not establish popularity-bias reduction). Partially conceded, now instrumented: the metric measures distillation fidelity to a composite teacher preference (revised Results/Discussion say this plainly). The 3-seed replication: alignment-only 68.8±4.7 > IPS 64.1±3.5 > single 62.3±3.7 > curriculum 60.6±10.3 — the seed-1 ranking does not replicate, and the stopping rule shows two new failure modes (single-stage transient dip; curriculum stage-boundary trap — patience fired at the data switch and never scanned stage 2, whose best evaluates at 57.7). Seed-1 significance: curriculum vs single-stage p=0.0023; vs IPS p=0.24 (n.s.).

W2 (missing baselines; probe confounds). SFT, DPO, and both rules are run (Q4); confounds instrumented (Q5). Single dataset remains the one gap we cannot close in this window — first item in the revised limitations.

Thank you for the candid response and thorough follow-up experiments.

I appreciate the authors reporting the new comparisons—particularly showing that standard SFT (90.94%) and even a simple attribute-overlap rule (78.86%) substantially outperform the proposed GRPO recipes, and that the multi-seed replication reverses the initial ranking of curriculum learning. While reframing the work around the diagnostic probe is sensible, the lack of demonstrated advantage for the main training method, combined with the single-dataset scope and missing full-ranking evaluations, leaves the contribution below the acceptance threshold.

Therefore, I will maintain my original rating

Reviewer_p78nOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to transfer a large language model's judgment about item popularity bias into a smaller sequential recommender. The authors construct real item pairs that are similar in attributes but differ substantially in popularity, use them as counterfactual sensitivity probes, and have a large teacher model produce preference labels. They then distill the teacher's judgments into a smaller student through single-stage GRPO training and two-stage curriculum learning. Experiments on the Amazon Beauty dataset evaluate preference-judgment accuracy, training stability, and computational cost across training strategies.

Critical strengths
  1. The motivation is reasonably novel. The paper approaches recommendation bias through a model's sensitivity to changes in popularity rather than through overall recommendation accuracy alone; by constructing real item pairs that are attribute-similar but popularity-different, the authors offer a diagnostic for whether a model relies excessively on popularity information.

  2. The paper analyzes practical training problems in reasonable detail. The authors identify and discuss candidate-position shortcuts, anomalous growth of output entropy, and stopping-rule failure, and address them with candidate-slot randomization, an adaptive KL constraint, and reward clipping. These observations are useful independently of the comparison, as they help explain instability of GRPO in recommendation.

  3. The results are reported candidly and include cost measurements. The paper does not avoid the unfavorable findings, namely that single-stage training underperforms, that curriculum learning involves clear trade-offs, and that the popularity heuristic remains stronger; it also measures teacher and student inference latency, throughput, and training wall-clock rather than assuming them.

Critical weaknesses
  1. The counterfactual sensitivity metric does not yet directly reflect popularity debiasing. As the authors note, the teacher selects the more popular item in most sensitivity samples, because attribute matching makes coverage values close and thereby triggers the teacher's popularity tie-breaker; the simple popularity rule also remains above every trained model on this metric. This explanation is reasonable, but it shows that the metric measures agreement with the teacher's composite preference: a student that actually learned to down-weight popularity could score lower. The authors should report results stratified by coverage gap and introduce a debiasing metric that is independent of the teacher's labels.

  2. The information gap between teacher and student may undermine the validity of the distillation target. The teacher directly receives popularity and history-attribute overlap statistics, whereas the student sees only attribute tags; when two items have similar or identical attributes, the student cannot access the evidence on which the teacher's judgment rests. The authors should quantify the share of such indistinguishable samples and clarify which teacher preference rules can be supported by the student's inputs.

  3. The construction of counterfactual samples still contains confounds. Matching some attributes while enlarging the popularity gap leaves items differing in attribute cardinality, item quality, and other unobserved factors, and popularity is not equivalent to exposure bias. The authors acknowledge these limitations, but no control experiments are provided, so the observed prediction differences cannot be fully attributed to popularity.

  4. The scale of the experiments and the baseline setup do not support the main conclusions. The main experiments use a single dataset, and each training strategy is run with a single random seed. Although curriculum learning attains the highest sensitivity accuracy among all trained methods, its alignment accuracy is below both training baselines, and its candidate-position swap consistency is only slightly above chance. Moreover, the full ranking evaluation with Recall and NDCG, although implemented, is not completed, and comparisons with more direct preference-distillation baselines such as SFT and DPO are missing. The advantage of curriculum over single-stage training is therefore still suggestive.

  5. Some terminology and efficiency claims should be stated more carefully. The paper calls attribute-matched comparisons between real items "counterfactual", which is not equivalent to a causal intervention that holds other factors fixed; its diagnostic meaning should be made explicit. Calling the popularity heuristic a performance "floor" is also misleading, since it actually outperforms every trained model on the sensitivity metric. On efficiency, the paper scopes the target to nearline re-ranking over a small candidate pool, yet a single pairwise judgment by the student is still on the order of seconds, while such re-ranking requires many pairwise calls; whether the end-to-end latency meets the target is not discussed.

Questions for authors

See Weaknesses

2 comments

Thank you for the precise reading — your five weaknesses map exactly onto what we ran during the discussion period. All numbers are new measurements on the same eval pairs (setup in the global response; full tables in the revised PDF). The headline changes: SFT dominates our GRPO recipes, and we now build the revision around that finding and the diagnostic framework.

W1: "…report results stratified by coverage gap and introduce a debiasing metric independent of the teacher's labels." Both done.

Teacher-independent metric. Each sensitivity pair contains the user's actual next item and its attribute-matched twin, so we measure P(model picks the actual next item) with no teacher label involved:

MethodTrue-next (95% CI)Pop-pick (TI)
Popularity-only79.19100
Alignment-only59.0657.05
IPS-weighted58.2551.85
Single-stage55.5651.18
Curriculum62.42 (56.8–67.7)58.39
SFT (new)74.50 (69.3–79.1)65.10
Teacher itself73.4970.81

Two readings on the record. (a) Students are not popularity imitators: pop-pick rates 50–65% vs the teacher's 70.8% and the heuristic's 100%. (b) Popularity is genuinely predictive in this probe distribution — the actual next item is the more-popular twin 79.19% of the time (positives are sampled by interaction frequency), which is exactly the benign-bias confound we cite (wang2023benign) and now quantify. Even SFT, which reproduces the teacher almost perfectly, does not beat the popularity heuristic on raw next-item prediction — "debiasing" on this log cannot and should not mean that.

Coverage-gap stratification (teacher-agreement):

StratumAlign-onlyIPSSingleCurriculumSFT
near-tie |Δcov≤0.1 (n=166)54.2250.0049.4060.84
discrepant (n=132)75.0082.5870.4578.7997.01

The decomposition confirms the mechanism: near-tie pairs are where the teacher's popularity tie-breaker dominates the labels. Curriculum is the most balanced GRPO variant across the two components; SFT learns both components.

W2: information gap between teacher and student. Now answered quantitatively, in both directions. The hard ceiling is small: only 13/298 pairs (4.4%) have literally identical attribute sets. And the input is far more sufficient than our crude rule suggested: SFT (which sees only attributes, same as the GRPO student) reaches 90.94% teacher-agreement and 74.5% true-next — i.e. the teacher's composite judgment is almost fully recoverable from attribute tags alone. The input supports ~90%+; our GRPO recipes extract only 59–69% of it. We state this plainly in the revision: the binding constraint was the training method, not the information gap.

W3: confounds (cardinality, quality, popularity≠exposure). Quality signals do not exist in this dataset — stated as a limitation. Cardinality is now controlled: on the 136/298 pairs with |A(i)|=|A(i′)|, curriculum 70.59 > align-only 62.50 > IPS 59.56 > single 55.15 (SFT 86.0) — the GRPO-method gaps widen on the clean subset, so the comparison does not rest on the confound. The confound itself is quantified inside the labels: on unequal-cardinality pairs the teacher picks the fewer-tag item 72.8% of the time.

W4: single dataset, single seed, missing Recall/NDCG, missing SFT/DPO.

  • SFT and DPO on the identical data: SFT 90.94 (see global response — it dominates our recipes), DPO 48.99 (fails). Reported as the central negative result.
  • Multi-seed (4 methods × 3 seeds, identical pre-registered rule): alignment-only 68.8±4.7 > IPS 64.1±3.5 > single 62.3±3.7 > curriculum 60.6±10.3 (seed-2 = rule-literal stage-1 checkpoint; stage-2 best 57.7). The seed-1 ranking does not replicate. Two new stopping-rule failure instances documented (single-stage transient-dip, seed 1; curriculum stage-boundary, seed 2).
  • Significance at seed 1, exact McNemar: curriculum vs single-stage p=0.0023 (agreement) / p=0.047 (true-next); curriculum vs IPS p=0.24 (n.s.). The revision softens "clears both baselines" accordingly.
  • Recall@10/NDCG@10 full-scale: not runnable in the window (1,225 LLM calls per user); remains the stated follow-up. Single dataset: first item in limitations.

W5: terminology and end-to-end latency. All accepted. "Floor" renamed (the popularity heuristic is an upper reference here); "counterfactual" explicitly scoped as a popularity-contrastive minimal-pair diagnostic, not a causal intervention (impossible without exposure randomization). E2E latency, measured on real prompts: one k=10 pool = C(10,2)=45 batched judgments — student ~2.5 s; teacher (32B, reasoning on) ~26.3 s (~10.5× per pool; consistent with Table-6 batched throughput, 45/1.89 ≈ 23.8 s). Added to the revised PDF so the nearline scoping is verifiable.

Thank you for the rebuttal. The authors have addressed my concerns, and I will maintain my original score.

Comments on the paper

1 comment

Dear Reviewers,

Thanks for your time and effort in reviewing this paper. The authors have responded to the comments by providing a rebuttal. Please read the rebuttal and indicate whether your concerns have been resolved. Thanks.

Regards from the AC.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.