acceptodds
Under review as a conference paper at ICLR 2027

RPSD: Retrieval-Privileged On-Policy Self-Distillation with Optimal Transport

Abstract

Sequential recommendation predicts a user's next interaction from historical behavior. Recent LLM-based methods recast this task as generative recommendation and increasingly adopt reinforcement learning such as GRPO for post-training. However, these methods suffer from two limitations, namely credit-blind optimization, where a sequence-level reward is shared across all generated tokens, and collaborative-blind supervision, where each training signal is derived from a single user and ignores behavioral patterns shared across similar users. To address both issues, we propose Retrieval-Privileged on-policy Self-Distillation (RPSD). For credit-blind optimization, we introduce on-policy self-distillation (OPSD), in which a teacher with privileged knowledge re-evaluates the student's on-policy response, and the token-level confidence difference between the student and teacher reshapes the shared sequence-level advantage into token-specific credit. However, the privileged knowledge in vanilla OPSD is still derived from a single user, leaving collaborative-blind supervision unresolved. We therefore ground the teacher in collaborative behavioral evidence from similar users. Specifically, we use temporally aware optimal transport (OT) to compare interaction trajectories across users by jointly considering item similarity and temporal structure, and further apply diversity-aware selection to retain complementary trajectories as privileged evidence. In this way, on-policy self-distillation provides fine-grained token-level credit, while OT enriches the teacher with structured cross-user evidence. To the best of our knowledge, this is the first work to bring on-policy self-distillation to sequential recommendation and to use cross-user behavior as its privileged signal. Extensive experiments across multiple real-world datasets demonstrate that RPSD achieves state-of-the-art performance.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Reinforcement Learning, Self-Distillation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_nJSNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Retrieval-Privileged On-Policy Self-Distillation (RPSD) to address two limitations of reinforcement learning for LLM-based sequential recommendation:

  1. Credit-blind optimization: Conventional GRPO assigns the same sequence-level advantage to all generated tokens, overlooking their different contributions to the final recommendation.
  2. Collaborative-blind supervision: Existing approaches primarily rely on individual users' interaction histories and ground-truth target items, without sufficiently exploiting collaborative behavioral patterns from similar users.

To address these issues, the authors introduce two complementary mechanisms.

First, RPSD employs On-Policy Self-Distillation (OPSD), where the student generates responses from the original recommendation prompt, while a teacher sharing the same model parameters additionally receives the ground-truth target item and retrieved behavioral evidence as privileged information. By comparing the teacher's and student's token-level log-probabilities on the same sampled responses, the method constructs advantage-aware token weights to redistribute the sequence-level GRPO advantage.

Second, the authors introduce temporally aware Optimal Transport (OT) to retrieve behavioral trajectories from similar users. The retrieval mechanism jointly considers semantic similarity and temporal alignment, followed by diversity-aware selection to obtain complementary behavioral evidence for the privileged teacher.

The teacher and retrieval components are used only during training, introducing no additional retrieval requirements at inference time.

Experiments on MovieLens-1M, MicroLens, and Netflix demonstrate that RPSD generally outperforms GRPO and other competitive sequential recommendation baselines across multiple ranking metrics.

Critical strengths

1. Reasonable and well-motivated methodological design.

The paper identifies two relevant limitations of GRPO-based sequential recommendation: uniform token-level credit assignment and insufficient utilization of collaborative behavioral information. Combining on-policy self-distillation with retrieval-based privileged supervision is a reasonable approach to addressing these limitations. The overall framework is coherent, and the design of using privileged information only during training is practically appealing, as it avoids introducing additional retrieval dependencies at inference time.

2. Competitive experimental performance.

The experimental results demonstrate consistent improvements over the GRPO baseline across three benchmark datasets. For example, in Table 1, RPSD improves NG@5 from 0.6495 to 0.7007 on MovieLens-1M and from 0.6198 to 0.7448 on Netflix. Under the larger candidate-set setting (100 candidates), RPSD also improves Netflix NG@10 from 0.1503 to 0.2466 compared with GRPO, as reported in Table 2. These results provide empirical support for the effectiveness of the overall framework.

Critical weaknesses

1. Limited methodological novelty, with the main contribution largely consisting of an incremental extension and combination of existing techniques.

My primary concern is the limited methodological novelty of the proposed framework.

The central idea of using a privileged teacher to re-evaluate the student's on-policy responses and provide token-level supervision is directly inherited from existing On-Policy Self-Distillation (OPSD) methods. The use of the same model under different conditioning contexts to obtain dense token-level guidance has already been established in prior work.

Compared with existing OPSD approaches, the main methodological modification in RPSD is the conversion of the teacher–student confidence gap into advantage-aware token weights, as described in Eq. (7). While this design is intuitively reasonable, it appears closer to a heuristic gradient-reweighting strategy than a fundamentally new credit-assignment principle.

In particular, the proposed token weights only rescale the magnitude of the original sequence-level advantage without changing its sign. Therefore, the method does not explicitly estimate the causal contribution of individual tokens to recommendation correctness. The paper also does not provide a theoretical analysis establishing that these confidence-based weights correspond to more accurate token-level credit assignment.

Furthermore, the retrieval component primarily combines established techniques, including semantic similarity, temporally aware optimal transport, and diversity-based selection.

The ablation results in Table 3 also show that the variant without retrieved evidence already achieves substantial improvements over GRPO. For instance, on MovieLens-1M, RPSD without OT achieves the same HR@5 (0.8716) and NG@5 (0.7007) as the full model.

Overall, although the combination is practically effective, the individual technical components largely build upon existing approaches, and the paper's methodological contribution appears relatively incremental.

2. The retrieval-privileged teacher does not fundamentally eliminate label-induced shortcut learning, and the evidence for genuinely retrieval-grounded supervision remains insufficient.

The paper explicitly acknowledges that a teacher conditioned on the ground-truth label may exploit label-induced shortcuts, producing supervision that reflects knowledge of the correct answer rather than a genuine understanding of the user's behavioral preferences.

Although RPSD introduces retrieved behavioral trajectories to mitigate this issue, the teacher still directly receives the ground-truth target item as part of its privileged input, as specified in Section 3.3.

Consequently, the teacher may still assign higher probabilities to tokens associated with the correct recommendation primarily because it already knows the answer, rather than because it has successfully identified meaningful collaborative patterns from the retrieved trajectories.

The ablation results in Table 3 demonstrate that retrieved evidence contributes to recommendation performance. For example, on Netflix, introducing retrieved evidence improves NG@5 from 0.6509 to 0.7448. This is an encouraging result indicating that retrieval is useful.

However, improved downstream performance does not necessarily demonstrate that the teacher's token-level supervision is genuinely grounded in the retrieved behavioral evidence rather than dominated by the ground-truth label.

The current experiments mainly evaluate recommendation accuracy and training dynamics, but provide limited direct evidence disentangling the respective contributions of label information and retrieved behavioral evidence to the teacher–student confidence gap.

More targeted analyses would strengthen this claim. For example, the authors could keep the ground-truth label fixed while perturbing or replacing the retrieved behavioral trajectories, and examine how the resulting token-level confidence gaps and advantage weights change. This would help determine whether the teacher actually relies on collaborative behavioral evidence rather than primarily exploiting the privileged label.

Therefore, although retrieval-based supervision improves empirical performance, the paper has not sufficiently established that its proposed mechanism fundamentally addresses the label-induced shortcut problem identified in prior OPSD research.

3. Insufficient justification for the necessity of Optimal Transport compared with alternative temporal sequence-matching methods.

Another important concern is whether Optimal Transport is genuinely necessary for constructing the proposed retrieval-based privileged evidence.

The authors motivate temporally aware OT by arguing that conventional cosine similarity ignores the temporal structure of user interactions, whereas OT can jointly capture semantic similarity and temporal alignment.

While this motivation is reasonable, the ability to model temporal dependencies is not unique to Optimal Transport. Several established sequence-matching approaches could potentially achieve similar objectives, including:

  • Transformer-based sequence representations, where interaction histories are encoded with positional or temporal information before computing similarities.
  • Dynamic Time Warping (DTW), which explicitly aligns sequences while accounting for temporal variations.
  • Temporal position-aware similarity measures that incorporate both item semantics and interaction order.

The paper provides ablation studies comparing the full model with variants that remove OT re-ranking or temporal alignment costs. For example, Table 4 shows that removing OT re-ranking decreases Netflix NG@5 from 0.7448 to 0.6404, while removing the temporal cost decreases it to 0.6393.

These results support the importance of the proposed retrieval design and its temporal information. However, they do not establish that OT is superior to other temporal sequence-matching techniques.

In particular, the "w/o Re-ranking" baseline relies on coarse cosine similarity over aggregated item embeddings, which is not a sufficiently strong temporal-aware alternative. Therefore, its inferior performance cannot be attributed specifically to the absence of optimal transport.

A more convincing evaluation would compare OT against alternative temporal matching strategies, such as DTW or Transformer-based sequence similarity, under comparable retrieval configurations.

Without such comparisons, it remains unclear whether the observed improvements originate from the specific properties of optimal transport or simply from incorporating temporal structure into the retrieval process.

Consequently, the methodological necessity and distinctive advantages of adopting OT are not sufficiently established by the current experiments.

Questions for authors

see weaknesses

6 comments
AuthorsOpenReview ↗

Rebuttal 1

We thank the reviewer for the constructive feedback. Our responses are as follows.


Q1. Limited methodological novelty, with the main contribution largely consisting of an incremental extension and combination of existing techniques.

A1 (Novelty). RPSD is not a combination of existing techniques, and our results show why. It is not OPSD with a retrieval module attached. OPSD does not transfer to recommendation out of the box, and making self-distillation work here required redesigning both the optimization and the privileged signal together, since the two are mutually dependent rather than stacked. The table below summarizes the differences, and we answer each point afterward.

AspectPrior RL for seq-rec (GRPO, Rank-GRPO)RLVR / OPSDRPSD (ours)
Credit granularitysequence-leveltoken-leveltoken-level
Privileged teachernonelabel onlycross-user behavior and label
Temporal and collaborative privilege✗✗✓
  1. Directly applying OPSD underperforms GRPO, motivating a different use of the teacher signal. Vanilla OPSD trains the student to match a label-conditioned teacher distribution. On MicroLens this gives HR@3 0.5669 against 0.6621 for plain GRPO (Table 6), so teacher-student self-distillation does not automatically help recommendation. RPSD instead uses the teacher-student gap to redistribute the magnitude of the reward-based advantage across tokens. The verifiable reward still sets the update direction, while the teacher decides which tokens receive more emphasis.
  2. The contribution is introducing this token-level credit mechanism into recommendation. Prior RL methods such as GRPO and Rank-GRPO share one sequence-level advantage across every token, which is credit-blind. RPSD is the first to bring token-level teacher-student credit to this task. We keep the reweighting rule on purpose, because the novelty is bringing this credit granularity, which recommendation RL lacked, into the field through a privileged teacher. A rule resembling prior self-distillation is expected, since the granularity is what we transfer.
  3. Preserving the advantage sign keeps token-level guidance anchored to the verifiable reward. The teacher sees the target label and retrieved evidence, but its probability estimates can still be imperfect, so RPSD lets it adjust the strength of an update while the reward fixes the direction. This is an intentional limit on the teacher's influence. We do not claim the weights estimate each token's causal contribution. Our claim is that differentiating token emphasis within the reward-based objective improves recommendation, and the ablation shows credit-blind optimization is the real bottleneck of GRPO, which our reweighting addresses.
  4. Our retrieval is a single integrated design, not a stack of off-the-shelf parts. It is a coarse-to-fine pipeline that carries temporal structure throughout. A recency-weighted history embedding recalls candidate windows, an unbalanced and temporally aware OT re-ranking aligns the two trajectories item by item under a joint semantic and position cost, and a diversity step keeps the evidence complementary. Time is built into the recall through recency weighting, the transport through the position cost, and the mass through recency-weighted marginals. To the best of our knowledge, no prior OT-based retrieval is built this way, since standard OT matching compares static sets with no coarse-to-fine stage, no temporal position cost, and no cross-user trajectory alignment. This purpose-built retrieval is the contribution.
  5. The ablation supports the design rather than undercutting it. That RPSD without OT already beats GRPO is our point, not a problem. It shows the token-level objective alone fixes the credit-blindness of GRPO, before any retrieval is added. The benefit of retrieval then grows with data sparsity, which is exactly how a collaborative signal should behave. The table below reports NDCG@3 for RPSD without OT and the full RPSD on all three datasets.
Dataset (avg history length)RPSD w/o OTRPSD fullRetrieval gain
MovieLens-1M (165)0.65930.6678small
MicroLens (8.8)0.58810.6036moderate
Netflix (5.2)0.58640.6952large

On the dense MovieLens-1M, each user's history is already long, so cross-user evidence adds little and the HR@5 and NG@5 the reviewer cites are already near the ceiling. That is saturation, not useless retrieval, and the finer HR@3 and NDCG@3 still improve. On the sparse Netflix, where histories average about five interactions, the own history is thin and cross-user evidence helps a lot. Reading the saturated metric on the densest dataset understates what retrieval does.

AuthorsOpenReview ↗

Rebuttal 2

Q2. The retrieval-privileged teacher does not fundamentally eliminate label-induced shortcut learning, and the evidence for genuinely retrieval-grounded supervision remains insufficient.

A2 (Retrieval-grounded supervision). We ran exactly the required experiment on Netflix (page 10; Table 6), and the gain disappears once the retrieved evidence is corrupted, even though the label is always present. We keep the ground-truth label fixed and corrupt only the retrieved trajectories, holding the block format and length constant, so the teacher label access is available across conditions and only the retrieved evidence changes.

Retrieved EvidenceHR@3NDCG@3HR@5NDCG@5
No evidence0.70900.58640.86600.6509
Random evidence0.70950.58140.87100.6435
Shuffled continuation0.72250.60580.86650.6558
OT (Ours)0.80200.69520.92450.7448
  1. The label alone does not drive the gain. Every row gives the teacher the same label, and only the evidence changes. Random evidence keeps the label but fills the block with unrelated users, and it stays at HR@3 0.7095 and NG@5 0.6435, close to the no-evidence row (0.7090 and 0.6509). So even with the label present, broken evidence performs like no evidence at all.
  2. The signal is the correct history-to-continuation mapping. Shuffled continuation keeps the genuinely retrieved histories but permutes their next item, and reaches only HR@3 0.7225, with its HR@5 and NG@5 at the no-evidence level, barely above random. The useful signal is therefore what a similar user actually did next, not the label and not the mere presence of similar histories. Only true OT evidence recovers the gain, reaching HR@3 0.8020 and NG@5 0.7448.
  3. Designing the label cannot create a shortcut in the update direction. The weight in Eq 7 rescales only the magnitude of the advantage while the sign is fixed by the verifiable reward, so a label-confident teacher can only reinforce a correctly rewarded token more, and it can never flip an update or reward a wrong completion.

Q3. Insufficient justification for the necessity of Optimal Transport compared with alternative temporal sequence-matching methods.

A3 (Necessity of OT over other temporal matching). We ran exactly the comparison requested, and OT beats all three temporal-aware alternatives under the same budget, so the gain is specific to OT and not to temporal structure in general. Under the same retrieval setup, the same candidate pool, the same window length, and the same number of retrieved trajectories, we changed only the matching function and tested the three temporal-aware alternatives the reviewer names. Results on Netflix are below (page 10; Table 5).

Re-rankingHR@3NDCG@3HR@5NDCG@5
DTW0.70550.57420.87000.6409
Temporal position-aware0.71500.58690.88850.6549
Transformer sequence encoder0.72450.59430.89000.6573
OT (ours)0.80200.69520.92450.7448

OT wins against all three alternatives. The strongest of them, the Transformer encoder, still trails OT by a wide margin, with HR@3 moving from 0.7245 to 0.8020 and NG@5 from 0.6573 to 0.7448. So the gain is specific to OT and does not come from simply adding temporal structure, which was the attribution the reviewer was concerned about.

What sets OT apart is the kind of match it makes, not whether it uses time. All three baselines see order but match rigidly. DTW forces each item onto a single counterpart along one monotonic path and the position-aware measure still pairs items one to one. The Transformer encoder returns only a single similarity score with no alignment, so it can call a neighbor close but cannot say which past items make it close. OT instead solves a soft unbalanced transport between the two item sequences, so one interaction can support several related ones while weak or irrelevant interactions carry little mass, and temporal order enters through a position penalty in the cost. This item-level coupling is what the teacher reads as evidence, which is why swapping in any of the three lowers every column above.

Dear Reviewer, we have added the experiments you requested (see our response above). As the discussion period ends soon, we would greatly appreciate it if you could check whether our response addresses your concerns. Thank you!

Reviewer_7BjLOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RPSD for LLM-based sequential recommendation. Temporally aware optimal transport retrieves similar users’ interaction histories and subsequent items. This evidence, together with the ground-truth target, forms a privileged context for a teacher that shares parameters with the student. The teacher rescores the student’s sampled responses, and token-level log-probability differences are transformed into weights for GRPO advantages. Retrieval and teacher computation are used only during training.

Critical strengths

Similar users’ histories and observed continuations provide additional information beyond the current user’s history. Restricting this information to training allows the deployed student to operate without retrieval or teacher computation.

RPSD improves over Qwen3-4B + GRPO across all reported metrics on the three datasets in the main comparison. On Netflix, HR@3 increases from 0.6690 to 0.8020. Results with 100 candidates on MovieLens and Netflix further support the method’s effectiveness.

The paper compares label-only and retrieval-augmented teacher contexts and examines reranking, temporal information, diversity selection, and weight processing. The response-level versus token-level comparison on Netflix keeps the teacher context and other training components fixed, providing meaningful evidence for token-specific weighting.

Critical weaknesses

The comparison with direct distillation is confounded by different teacher information. Vanilla OPSD uses a label-conditioned teacher, whereas RPSD additionally provides retrieved behavioral evidence and changes the training objective. This comparison therefore does not establish whether advantage reweighting is preferable to direct distribution matching under the same teacher information. The retrieval and granularity ablations are informative, but do not resolve this distinction.

The interpretation of the weights as recommendation-relevant token credit is not directly established. The teacher receives the target item explicitly, while the response primarily consists of an item ID and title. The probability differences may therefore partly reflect access to the answer text rather than grounding in behavioral evidence. The performance gains and granularity ablation support the usefulness of the weighting scheme, but do not establish the stronger claim that it identifies tokens carrying recommendation evidence.

The training and data protocols lack important details. The paper does not adequately describe how the SFT checkpoint was trained or how its training instances relate to those used for RPSD. The instance-level split also leaves temporal ordering and potential overlap between histories unclear. Although the retrieval pool is described as using training sequences, its construction relative to the instance split requires clarification. These omissions limit reproducibility.

Configuration and training costs are incompletely reported. Numerical settings for several retrieval and weighting parameters, including the retrieval window length and guidance strength, are missing. The algorithm returns no retrieved trajectory for histories shorter than the window, but the prevalence of this case is not reported, leaving evidence coverage for short-history users unclear. The additional retrieval and teacher-scoring costs are also not quantified.

Questions for authors

How was the SFT checkpoint trained, and how do its training data relate to the RPSD training instances? How are temporal boundaries and overlapping histories handled when splitting instances and constructing the retrieval pool?

Given the differences in teacher information, how much of the improvement over vanilla OPSD can be attributed to the proposed optimization objective based on the existing evidence?

6 comments
AuthorsOpenReview ↗

Rebuttal 1

We sincerely thank the reviewer for the constructive feedback. Our responses are as follows.


Q1. The comparison with direct distillation is confounded by different teacher information. Vanilla OPSD uses a label-conditioned teacher, whereas RPSD additionally provides retrieved behavioral evidence and changes the training objective. This comparison therefore does not establish whether advantage reweighting is preferable to direct distribution matching under the same teacher information. The retrieval and granularity ablations are informative, but do not resolve this distinction.

A1 (Teacher information). The controlled comparison the reviewer asks for is already available from two results in the paper. Vanilla OPSD (Table 6) and RPSD w/o OT (Table 3) use the same label-only teacher and differ only in the objective, that is direct distribution matching versus advantage reweighting. RPSD w/o OT and the full RPSD then use the same advantage-reweighting objective and differ only in the teacher information. We combine them below on MicroLens.

VariantTeacher informationObjectiveHR@3HR@5NDCG@3NDCG@5
OPSDlabel onlydistribution matching0.56690.69420.48090.5404
RPSD w/o OTlabel onlyadvantage reweighting0.66720.78010.58810.6360
RPSD (full)label and retrieved evidenceadvantage reweighting0.68360.78090.60360.6417

Row one to row two fixes the teacher at label only and changes only the objective. Row two to row three fixes the objective and adds only the evidence. The gain over OPSD therefore comes first from the objective and then from the evidence.


Q2. The interpretation of the weights as recommendation-relevant token credit is not directly established. The teacher receives the target item explicitly, while the response primarily consists of an item ID and title. The probability differences may therefore partly reflect access to the answer text rather than grounding in behavioral evidence. The performance gains and granularity ablation support the usefulness of the weighting scheme, but do not establish the stronger claim that it identifies tokens carrying recommendation evidence.

A2 (Effect of weights). We establish a new RL training regime rather than selecting decisive tokens. The weights play two roles.

  1. The weights tell the model where to learn more. GRPO gives every token in a response the same advantage. RPSD uses the teacher-student probability gap to adjust how strongly each token is updated, while the reward still determines whether it is encouraged or discouraged. We do not claim that each weight directly measures how much behavioral evidence a token carries. Rather, the weights help distribute the learning signal within a response. The comparison with RPSD w/o OT keeps the teacher's access to the answer unchanged and supports the additional benefit of behavioral evidence.
  2. The weights change as the student learns. If the student is still uncertain about the output format, formatting tokens can also receive guidance. As the student learns the format, teacher and student may assign similar probabilities to those tokens, making their raw gaps smaller. The remaining differences can then give more guidance to other tokens, including those that identify the item. This explains how the method can follow the student's changing learning needs. It does not require every high-weight token to have a separate interpretation as behavioral evidence.
AuthorsOpenReview ↗

Rebuttal 2

Q3. The training and data protocols lack important details. The paper does not adequately describe how the SFT checkpoint was trained or how its training instances relate to those used for RPSD. The instance-level split also leaves temporal ordering and potential overlap between histories unclear. Although the retrieval pool is described as using training sequences, its construction relative to the instance split requires clarification. These omissions limit reproducibility.

A3 (Protocol and leakage). Datasets are split at the user level, not the interaction level, so no history is shared across splits and there is no temporal boundary ambiguity. Here we provide more details.

Data split. Different from Amazon datasets, MovieLens, MicroLens and Netflix are split at the user level into disjoint train, validation, and test user sets in a 7:1:2 ratio, not at the instance level. Each user therefore belongs to exactly one split, so no user history is shared across splits and there is no temporal boundary to define within a user between training and test interactions. This removes the overlap and boundary ambiguity raised in the review. Within a user, interactions follow the timestamp order provided by the dataset, as in prior multimodal sequential recommendation work.

DatasetTrain + Valid usersTest usersTotalTest ratioTrain/Test overlap
MovieLens-1M48311208603920.0%0
MicroLens54181337675519.8%0
Netflix78052000980520.4%0

SFT. SFT adapts the pretrained Qwen3-4B into a recommendation-oriented instruction follower by maximizing L_SFT = −E log π_φ(y|x) on the training split in the student-prompt format, trained for 3000 steps. RPSD then initializes from this checkpoint and post-trains on the same training-split instances, so SFT and RPSD use identical users, instances, and prompt format and differ only in the objective.

Retrieval pool. The pool is built only from training users, and at query time the retriever excludes the query user's own windows and the identical current window.


Q4. Configuration and training costs are incompletely reported. Numerical settings for several retrieval and weighting parameters, including the retrieval window length and guidance strength, are missing. The algorithm returns no retrieved trajectory for histories shorter than the window, but the prevalence of this case is not reported, leaving evidence coverage for short-history users unclear. The additional retrieval and teacher-scoring costs are also not quantified.

A4 (Configuration and training costs). We report the full parameter settings, evidence coverage, and measured training cost below.

ModuleParameterValue
Retrievalwindow length K9 (MovieLens-1M), 4 (MicroLens), 3 (Netflix)
Retrievalevidence coverage99.9% (MovieLens-1M), 100% (MicroLens), 97.41% (Netflix)
Retrievalrecall size N₀200
Retrievalrecency rate β0.23
Retrievalentropic regularization ε0.05
Retrievalmarginal relaxation ρ1.0
Retrievalposition cost γ0.1
RetrievalOT temperature τ0.1
Retrievaldiversity weight λ_div0.05
Retrievalretrieved windows m8
Weightingguidance strength λ0.2
Weightingweight bound ε_w0.05
Weightingdelta clip c_δ5.0
Traininggroup size G8
Traininglearning rate8e-6
Trainingepochs3
Trainingbatch × grad-accum4 × 8
TrainingKL weight β_KL0.1
Datacandidate size n_c10
Dataitem embedding dim128 (PCA)

On coverage. K is set near each dataset's history floor, so nearly all evaluated users have at least K history items and the no-trajectory fallback almost never fires. Evidence coverage is reported in the table above.

Training cost (RPSD, Netflix, 3 epochs).

ItemValue
Total training time8.7 hours (31,275 s)
Training steps1461
Time per step~21 s (optimization step 19.9 s)
Throughput0.75 samples/s
Tokens processed63.1 M
Completion length (mean / min / max)15.0 / 11 / 29 tokens
Clipped ratio0

On cost. The overall training cost is reported in the cost table above. The evidence index is built once before training. Retrieval re-ranks a recall list of 200 windows per instance with numpy on 128-dimensional embeddings and no gradient, so its cost is small and confined to training. Teacher scoring is one extra forward pass over the sampled completions per step with no generation, and the completions are short at a mean length of 15 tokens over G = 8 samples, so RPSD adds a single scoring pass over GRPO during training. The full run takes 8.7 hours for 3 epochs on Netflix. At inference the model uses only the student prompt, so there is no added retrieval or teacher cost.

Dear Reviewer, we have added the experiments you requested (see our response above). As the discussion period ends soon, we would greatly appreciate it if you could check whether our response addresses your concerns. Thank you!

Reviewer_Y8QTOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes RPSD, a retrieval-privileged on-policy self-distillation framework for LLM-based sequential recommendation. It addresses two limitations of GRPO: uniform token-level credit assignment and the lack of cross-user behavioral evidence.

Critical strengths
  1. The paper considers an interesting direction: using collaborative evidence available during training to improve the supervision of a generative recommender without increasing inference-time requirements.

  2. Encouraging experimental results with useful ablations. RPSD consistently improves over its GRPO backbone across the three datasets, with particularly notable gains on Netflix. The experimental section includes ablations of retrieval components, comparisons with label-only privileged supervision, and additional evaluations of token-level versus response-level guidance.

Critical weaknesses

W1. The methodological novelty is limited, particularly in the token-level credit assignment mechanism. The central token-level optimization closely resembles the existing Self-Distilled RLVR.

W2. Inconsistent retrieval benefits. Table 3 shows that removing retrieved evidence has almost no effect on HR@5 for MovieLens-1M and MicroLens, while the improvement on Netflix is much larger. The paper provides limited analysis of why retrieval contributes differently across datasets.

W3. Insufficient retrieval baselines. Although the paper ablates individual retrieval components, comparisons with simpler sequence-matching methods, such as DTW or position-aware similarity, are missing. This makes it difficult to establish whether the proposed OT-based retrieval is necessary.

W4. Limited validation of retrieved evidence quality. The paper lacks controls using randomly selected trajectories or shuffled trajectory continuations. Such experiments would help determine whether the improvements come from meaningful collaborative evidence rather than simply providing additional context to the teacher.

W5. The evaluation primarily uses small, randomly sampled candidate sets (10 or 100 items), limiting its relevance to realistic recommendation settings.

Questions for authors

While the temporally aware OT retrieval pipeline is reasonable, its components largely build on established techniques. The current ablations demonstrate the benefit of OT re-ranking but do not establish its advantage over simpler sequence-alignment methods. Could the authors compare against alternatives such as DTW or position-aware similarity matching under comparable retrieval budgets? This would help clarify whether OT is necessary for constructing effective privileged evidence.

2 comments
AuthorsOpenReview ↗

Rebuttal 1

We thank the reviewer for the constructive feedback. Our responses are as follows.


Q1.The methodological novelty is limited, particularly in the token-level credit assignment mechanism. The central token-level optimization closely resembles the existing Self-Distilled RLVR.

A1 (Novelty). Our contribution is not a new token rule. It is two things. First, we are the first to bring token-level teacher-student distillation into sequential recommendation, which has so far used only sequence-level advantage. Second, we design a privileged teacher built for the task rather than plugging in RLVR and OT.

  1. The shared rule is deliberate. GRPO and Rank-GRPO assign one sequence-level advantage to every token, which is credit-blind. We transfer exactly the token-level granularity that recommendation RL was missing, so a rule resembling RLVR is expected. The contribution is the transfer and making it work, not the rule.
  2. It is not RLVR ported to a new domain. A direct transfer fails. Vanilla OPSD underperforms even GRPO (Table 6), because a label-only teacher is thin in recommendation and gives the rule almost nothing to redistribute. We therefore design a recommendation-specific privilege, namely cross-user behavior retrieved by temporally aware OT.
  3. It is not a concatenation of RLVR and OT. The two are mutually dependent. The credit rule is inert without an informative teacher, and the retrieval is what makes the teacher informative. The table below makes the difference from Self-Distilled RLVR concrete.
AspectPrior RL for seq-rec (GRPO, Rank-GRPO)RLVR / OPSDRPSD (ours)
Credit granularitysequence-leveltoken-leveltoken-level
Privileged teachernonelabel onlycross-user behavior and label
Temporal and collaborative privilege✗✗✓

Q2. Inconsistent retrieval benefits. Table 3 shows that removing retrieved evidence has almost no effect on HR@5 for MovieLens-1M and MicroLens, while the improvement on Netflix is much larger. The paper provides limited analysis of why retrieval contributes differently across datasets.

A2 (Dataset-dependent gains). Retrieved evidence helps on all three datasets, and its size tracks dataset density. The gains from adding retrieval (RPSD Full over w/o OT) are shown below.

DatasetHR@3NDCG@3HR@5
MovieLens-1M+2.3+0.9+0.0
MicroLens+1.6+1.6+0.1
Netflix+9.3+10.9+5.9

On MovieLens and MicroLens the effect looks flat only on HR@5, which is already near-saturated at 0.87 and 0.78, so the gain surfaces in the finer HR@3 and NDCG@3. On the sparse Netflix dataset, where the average history is about 5.23 interactions versus about 165 on MovieLens, a user's own history is far less informative and cross-user evidence contributes much more. Appendix H confirms this through the teacher-student confidence gap, which stays small on the dense datasets and widens on Netflix.


Q3. Insufficient retrieval baselines. Although the paper ablates individual retrieval components, comparisons with simpler sequence-matching methods, such as DTW or position-aware similarity, are missing. This makes it difficult to establish whether the proposed OT-based retrieval is necessary.

A3 (DTW and position-aware baselines). We added three re-ranking baselines under the same retrieval budget, and OT wins on every metric, so the gain comes from OT alignment rather than from retrieval alone or a generic alignment rule. All baselines share the same coarse recall, the same leakage guards, the same diversity selection, and the same title embeddings, and they differ only in the re-ranking operator. DTW and the position-aware baseline use the same semantic and position ground cost as OT, so only the alignment operator changes. The Transformer baseline encodes each ordered history with the same frozen SFT backbone. Results on Netflix are shown below (page 10; Table 5).

Re-rankingHR@3NDCG@3HR@5NDCG@5
DTW0.70550.57420.87000.6409
Temporal position-aware0.71500.58690.88850.6549
Transformer sequence encoder0.72450.59430.89000.6573
OT (ours)0.80200.69520.92450.7448

This matches Section 4.3, where poorly matched evidence does worse than using no evidence. The larger NDCG gap also shows that the OT evidence mainly helps ranking.

AuthorsOpenReview ↗

Rebuttal 2

Q4. Limited validation of retrieved evidence quality. The paper lacks controls using randomly selected trajectories or shuffled trajectory continuations. Such experiments would help determine whether the improvements come from meaningful collaborative evidence rather than simply providing additional context to the teacher.

A4 (Evidence-quality controls). The gain comes from the correct history-to-next-item mapping, not from extra context. We added two controls (page 10; Table 6) that keep the evidence block's format and length fixed but destroy its content. The first uses random trajectories from unrelated users, and the second uses shuffled continuations where each retrieved history is paired with another user's next item.

Retrieved EvidenceHR@3NDCG@3HR@5NDCG@5
No evidence0.70900.58640.86600.6509
Random evidence0.70950.58140.87100.6435
Shuffled continuation0.72250.60580.86650.6558
OT (Ours)0.80200.69520.92450.7448

All four rows use the same teacher and label, so only the evidence content differs. Random evidence stays at the no-evidence level, so extra context alone does nothing. Shuffled continuation, which keeps the retrieved histories but scrambles their next item, barely moves. Only genuine OT evidence rises well above both.


Q5. The evaluation primarily uses small, randomly sampled candidate sets (10 or 100 items), limiting its relevance to realistic recommendation settings.

A5 (Candidate set size). The sampled-candidate protocol is standard practice in LLM-based recommendation, not a choice specific to this paper. A small candidate set of one ground-truth item with sampled negatives is the standard evaluation in LLM-based recommenders such as LLaRA [1], S-DPO [2], and SPRec [3]. We use the same construction, and every baseline in our tables is evaluated under the identical candidate set and data split, so the comparison is fair. We also report the more challenging n_c=100 setting in Table 2, where RPSD remains effective as the candidate pool grows.


References

[1] Jiayi Liao, Sihang Li, Zhengyi Yang, Jiancan Wu, Yancheng Yuan, Xiang Wang, and Xiangnan He. 2024. LLaRA: Large Language-Recommendation Assistant. In SIGIR. 1785-1795.

[2] Yuxin Chen, Junfei Tan, An Zhang, Zhengyi Yang, Leheng Sheng, Enzhi Zhang, Xiang Wang, and Tat-Seng Chua. 2024. On Softmax Direct Preference Optimization for Recommendation. In NeurIPS.

[3] Chongming Gao, Ruijun Chen, Shuai Yuan, Kexin Huang, Yuanqing Yu, and Xiangnan He. 2025. SPRec: Self-Play to Debias LLM-based Recommendation. In WWW.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.