- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Retrieval-Privileged On-Policy Self-Distillation (RPSD) to address two limitations of reinforcement learning for LLM-based sequential recommendation:
- Credit-blind optimization: Conventional GRPO assigns the same sequence-level advantage to all generated tokens, overlooking their different contributions to the final recommendation.
- Collaborative-blind supervision: Existing approaches primarily rely on individual users' interaction histories and ground-truth target items, without sufficiently exploiting collaborative behavioral patterns from similar users.
To address these issues, the authors introduce two complementary mechanisms.
First, RPSD employs On-Policy Self-Distillation (OPSD), where the student generates responses from the original recommendation prompt, while a teacher sharing the same model parameters additionally receives the ground-truth target item and retrieved behavioral evidence as privileged information. By comparing the teacher's and student's token-level log-probabilities on the same sampled responses, the method constructs advantage-aware token weights to redistribute the sequence-level GRPO advantage.
Second, the authors introduce temporally aware Optimal Transport (OT) to retrieve behavioral trajectories from similar users. The retrieval mechanism jointly considers semantic similarity and temporal alignment, followed by diversity-aware selection to obtain complementary behavioral evidence for the privileged teacher.
The teacher and retrieval components are used only during training, introducing no additional retrieval requirements at inference time.
Experiments on MovieLens-1M, MicroLens, and Netflix demonstrate that RPSD generally outperforms GRPO and other competitive sequential recommendation baselines across multiple ranking metrics.
Critical strengths
1. Reasonable and well-motivated methodological design.
The paper identifies two relevant limitations of GRPO-based sequential recommendation: uniform token-level credit assignment and insufficient utilization of collaborative behavioral information. Combining on-policy self-distillation with retrieval-based privileged supervision is a reasonable approach to addressing these limitations. The overall framework is coherent, and the design of using privileged information only during training is practically appealing, as it avoids introducing additional retrieval dependencies at inference time.
2. Competitive experimental performance.
The experimental results demonstrate consistent improvements over the GRPO baseline across three benchmark datasets. For example, in Table 1, RPSD improves NG@5 from 0.6495 to 0.7007 on MovieLens-1M and from 0.6198 to 0.7448 on Netflix. Under the larger candidate-set setting (100 candidates), RPSD also improves Netflix NG@10 from 0.1503 to 0.2466 compared with GRPO, as reported in Table 2. These results provide empirical support for the effectiveness of the overall framework.
Critical weaknesses
1. Limited methodological novelty, with the main contribution largely consisting of an incremental extension and combination of existing techniques.
My primary concern is the limited methodological novelty of the proposed framework.
The central idea of using a privileged teacher to re-evaluate the student's on-policy responses and provide token-level supervision is directly inherited from existing On-Policy Self-Distillation (OPSD) methods. The use of the same model under different conditioning contexts to obtain dense token-level guidance has already been established in prior work.
Compared with existing OPSD approaches, the main methodological modification in RPSD is the conversion of the teacher–student confidence gap into advantage-aware token weights, as described in Eq. (7). While this design is intuitively reasonable, it appears closer to a heuristic gradient-reweighting strategy than a fundamentally new credit-assignment principle.
In particular, the proposed token weights only rescale the magnitude of the original sequence-level advantage without changing its sign. Therefore, the method does not explicitly estimate the causal contribution of individual tokens to recommendation correctness. The paper also does not provide a theoretical analysis establishing that these confidence-based weights correspond to more accurate token-level credit assignment.
Furthermore, the retrieval component primarily combines established techniques, including semantic similarity, temporally aware optimal transport, and diversity-based selection.
The ablation results in Table 3 also show that the variant without retrieved evidence already achieves substantial improvements over GRPO. For instance, on MovieLens-1M, RPSD without OT achieves the same HR@5 (0.8716) and NG@5 (0.7007) as the full model.
Overall, although the combination is practically effective, the individual technical components largely build upon existing approaches, and the paper's methodological contribution appears relatively incremental.
2. The retrieval-privileged teacher does not fundamentally eliminate label-induced shortcut learning, and the evidence for genuinely retrieval-grounded supervision remains insufficient.
The paper explicitly acknowledges that a teacher conditioned on the ground-truth label may exploit label-induced shortcuts, producing supervision that reflects knowledge of the correct answer rather than a genuine understanding of the user's behavioral preferences.
Although RPSD introduces retrieved behavioral trajectories to mitigate this issue, the teacher still directly receives the ground-truth target item as part of its privileged input, as specified in Section 3.3.
Consequently, the teacher may still assign higher probabilities to tokens associated with the correct recommendation primarily because it already knows the answer, rather than because it has successfully identified meaningful collaborative patterns from the retrieved trajectories.
The ablation results in Table 3 demonstrate that retrieved evidence contributes to recommendation performance. For example, on Netflix, introducing retrieved evidence improves NG@5 from 0.6509 to 0.7448. This is an encouraging result indicating that retrieval is useful.
However, improved downstream performance does not necessarily demonstrate that the teacher's token-level supervision is genuinely grounded in the retrieved behavioral evidence rather than dominated by the ground-truth label.
The current experiments mainly evaluate recommendation accuracy and training dynamics, but provide limited direct evidence disentangling the respective contributions of label information and retrieved behavioral evidence to the teacher–student confidence gap.
More targeted analyses would strengthen this claim. For example, the authors could keep the ground-truth label fixed while perturbing or replacing the retrieved behavioral trajectories, and examine how the resulting token-level confidence gaps and advantage weights change. This would help determine whether the teacher actually relies on collaborative behavioral evidence rather than primarily exploiting the privileged label.
Therefore, although retrieval-based supervision improves empirical performance, the paper has not sufficiently established that its proposed mechanism fundamentally addresses the label-induced shortcut problem identified in prior OPSD research.
3. Insufficient justification for the necessity of Optimal Transport compared with alternative temporal sequence-matching methods.
Another important concern is whether Optimal Transport is genuinely necessary for constructing the proposed retrieval-based privileged evidence.
The authors motivate temporally aware OT by arguing that conventional cosine similarity ignores the temporal structure of user interactions, whereas OT can jointly capture semantic similarity and temporal alignment.
While this motivation is reasonable, the ability to model temporal dependencies is not unique to Optimal Transport. Several established sequence-matching approaches could potentially achieve similar objectives, including:
- Transformer-based sequence representations, where interaction histories are encoded with positional or temporal information before computing similarities.
- Dynamic Time Warping (DTW), which explicitly aligns sequences while accounting for temporal variations.
- Temporal position-aware similarity measures that incorporate both item semantics and interaction order.
The paper provides ablation studies comparing the full model with variants that remove OT re-ranking or temporal alignment costs. For example, Table 4 shows that removing OT re-ranking decreases Netflix NG@5 from 0.7448 to 0.6404, while removing the temporal cost decreases it to 0.6393.
These results support the importance of the proposed retrieval design and its temporal information. However, they do not establish that OT is superior to other temporal sequence-matching techniques.
In particular, the "w/o Re-ranking" baseline relies on coarse cosine similarity over aggregated item embeddings, which is not a sufficiently strong temporal-aware alternative. Therefore, its inferior performance cannot be attributed specifically to the absence of optimal transport.
A more convincing evaluation would compare OT against alternative temporal matching strategies, such as DTW or Transformer-based sequence similarity, under comparable retrieval configurations.
Without such comparisons, it remains unclear whether the observed improvements originate from the specific properties of optimal transport or simply from incorporating temporal structure into the retrieval process.
Consequently, the methodological necessity and distinctive advantages of adopting OT are not sufficiently established by the current experiments.
Questions for authors
see weaknesses
6 comments
Rebuttal 2
Q2. The retrieval-privileged teacher does not fundamentally eliminate label-induced shortcut learning, and the evidence for genuinely retrieval-grounded supervision remains insufficient.
A2 (Retrieval-grounded supervision). We ran exactly the required experiment on Netflix (page 10; Table 6), and the gain disappears once the retrieved evidence is corrupted, even though the label is always present. We keep the ground-truth label fixed and corrupt only the retrieved trajectories, holding the block format and length constant, so the teacher label access is available across conditions and only the retrieved evidence changes.
| Retrieved Evidence | HR@3 | NDCG@3 | HR@5 | NDCG@5 |
|---|---|---|---|---|
| No evidence | 0.7090 | 0.5864 | 0.8660 | 0.6509 |
| Random evidence | 0.7095 | 0.5814 | 0.8710 | 0.6435 |
| Shuffled continuation | 0.7225 | 0.6058 | 0.8665 | 0.6558 |
| OT (Ours) | 0.8020 | 0.6952 | 0.9245 | 0.7448 |
- The label alone does not drive the gain. Every row gives the teacher the same label, and only the evidence changes. Random evidence keeps the label but fills the block with unrelated users, and it stays at HR@3 0.7095 and NG@5 0.6435, close to the no-evidence row (0.7090 and 0.6509). So even with the label present, broken evidence performs like no evidence at all.
- The signal is the correct history-to-continuation mapping. Shuffled continuation keeps the genuinely retrieved histories but permutes their next item, and reaches only HR@3 0.7225, with its HR@5 and NG@5 at the no-evidence level, barely above random. The useful signal is therefore what a similar user actually did next, not the label and not the mere presence of similar histories. Only true OT evidence recovers the gain, reaching HR@3 0.8020 and NG@5 0.7448.
- Designing the label cannot create a shortcut in the update direction. The weight in Eq 7 rescales only the magnitude of the advantage while the sign is fixed by the verifiable reward, so a label-confident teacher can only reinforce a correctly rewarded token more, and it can never flip an update or reward a wrong completion.
Q3. Insufficient justification for the necessity of Optimal Transport compared with alternative temporal sequence-matching methods.
A3 (Necessity of OT over other temporal matching). We ran exactly the comparison requested, and OT beats all three temporal-aware alternatives under the same budget, so the gain is specific to OT and not to temporal structure in general. Under the same retrieval setup, the same candidate pool, the same window length, and the same number of retrieved trajectories, we changed only the matching function and tested the three temporal-aware alternatives the reviewer names. Results on Netflix are below (page 10; Table 5).
| Re-ranking | HR@3 | NDCG@3 | HR@5 | NDCG@5 |
|---|---|---|---|---|
| DTW | 0.7055 | 0.5742 | 0.8700 | 0.6409 |
| Temporal position-aware | 0.7150 | 0.5869 | 0.8885 | 0.6549 |
| Transformer sequence encoder | 0.7245 | 0.5943 | 0.8900 | 0.6573 |
| OT (ours) | 0.8020 | 0.6952 | 0.9245 | 0.7448 |
OT wins against all three alternatives. The strongest of them, the Transformer encoder, still trails OT by a wide margin, with HR@3 moving from 0.7245 to 0.8020 and NG@5 from 0.6573 to 0.7448. So the gain is specific to OT and does not come from simply adding temporal structure, which was the attribution the reviewer was concerned about.
What sets OT apart is the kind of match it makes, not whether it uses time. All three baselines see order but match rigidly. DTW forces each item onto a single counterpart along one monotonic path and the position-aware measure still pairs items one to one. The Transformer encoder returns only a single similarity score with no alignment, so it can call a neighbor close but cannot say which past items make it close. OT instead solves a soft unbalanced transport between the two item sequences, so one interaction can support several related ones while weak or irrelevant interactions carry little mass, and temporal order enters through a position penalty in the cost. This item-level coupling is what the teacher reads as evidence, which is why swapping in any of the three lowers every column above.
Reply
Dear Reviewer, we have added the experiments you requested (see our response above). As the discussion period ends soon, we would greatly appreciate it if you could check whether our response addresses your concerns. Thank you!
Rebuttal 1
We thank the reviewer for the constructive feedback. Our responses are as follows.
Q1. Limited methodological novelty, with the main contribution largely consisting of an incremental extension and combination of existing techniques.
A1 (Novelty). RPSD is not a combination of existing techniques, and our results show why. It is not OPSD with a retrieval module attached. OPSD does not transfer to recommendation out of the box, and making self-distillation work here required redesigning both the optimization and the privileged signal together, since the two are mutually dependent rather than stacked. The table below summarizes the differences, and we answer each point afterward.
On the dense MovieLens-1M, each user's history is already long, so cross-user evidence adds little and the HR@5 and NG@5 the reviewer cites are already near the ceiling. That is saturation, not useless retrieval, and the finer HR@3 and NDCG@3 still improve. On the sparse Netflix, where histories average about five interactions, the own history is thin and cross-user evidence helps a lot. Reading the saturated metric on the densest dataset understates what retrieval does.