acceptodds
Under review as a conference paper at ICLR 2027

Reasoning to Rank: Exploiting Large Language Models for Recommendation

Abstract

Recommender systems are tasked to infer users' evolving preferences and rank items aligned with their intents, which calls for in-depth reasoning beyond pattern-based scoring. Recent efforts start to leverage large language models (LLMs) for recommendation, but how to effectively optimize the model for improved recommendation utility is still under explored. In this work, we propose Reasoning to Rank, an end-to-end training framework that internalizes recommendation utility optimization into the learning of step-by-step reasoning in LLMs. To avoid position bias in LLM reasoning and enable direct optimization of the reasoning process, our framework performs reasoning at the user-item level and employs reinforcement learning for end-to-end training of the LLM. Experiments on three Amazon datasets and a large-scale industrial dataset showed consistent gains over strong conventional and LLM-based solutions. Extensive in-depth analyses validate the necessity of the key components in the proposed framework and shed lights on the future developments of this line of work.

Primary area:
foundation or frontier models, including LLMs
Keywords:
recommendation, large language models, reasoning, learning to rank

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_x8T2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

R2Rank is an LLM-based recommender in three parts.

  1. It generates a structured rationale independently for each user–candidate pair (Eq. 1).
  2. It maps the last-token hidden state of that rationale to a scalar score with an MLP head (Eq. 2).
  3. It trains the backbone and head end to end with a listwise reward. A Plackett–Luce prefix of length 10 is sampled from the scores, and its NDCG@10 serves as a shared return for two terms: a REINFORCE term on the PL log-probability, which flows through the scores into the backbone, and a clipped PPO term on the rationale tokens with a zero baseline (Eqs. 4–7, Alg. 1).

A self-check SFT warm start distilled from DeepSeek-R1 comes first (Sec. 3.4, App. A). The evaluation uses Qwen2.5-1.5B/3B on three Amazon datasets and a proprietary advertising dataset. Each user gets a fixed candidate set of one positive and 19 uniformly random negatives. R2Rank-3B beats all trainable same-backbone LLM baselines on NDCG@10 across the four datasets, with the largest margin on Industrial (0.818 vs 0.560 for LatentR3, Table 2). The paper adds ablations (Tables 3–6), rationale substitution and transfer with frozen checkpoints (Tables 7–8), and further analyses (App. D).

Critical strengths
  1. A sensible design that ties candidate-wise reasoning to a list-level objective. Each candidate is assessed in its own prompt, which removes candidate presentation order and the position bias of listwise LLM rankers (§3.2, App. D.4). The rationale's last-token hidden state is then scored, and the resulting scores are trained end to end with a Plackett–Luce surrogate of NDCG. The two gradient paths (PL scores into backbone and head; PPO on rationale tokens) are cleanly derived and clearly described (Eq. 6, Alg. 1). Within the evaluated protocol, R2Rank outperforms all same-backbone LLM baselines on NDCG@10 (Table 2), although Weakness 2 limits how far this comparison can be read.
  2. Rationale-content controls that go beyond illustrative examples. With frozen checkpoints and fixed final answers, original rationales outperform rationales shuffled within the same user on all four datasets. On three of the four, shuffled rationales even fall below answer-only completions (Table 7). R2Rank rationales also improve DeepSeek-R1 from 0.512 to 0.552 (Table 8). This is non-trivial evidence that the scorer uses candidate-specific rationale content, not just extra tokens, and the paper is careful not to overclaim faithfulness.
Critical weaknesses
  1. The central claim, optimizing the reasoning process itself for ranking utility, is not isolated. The model trained without CoT already beats every baseline (Musical 0.545 vs 0.512; Industrial 0.758 vs 0.560; Table 5 vs Table 2), and on Musical plain CoT (0.494) is worse than no CoT (Table 6). No ablation removes the PPO term while keeping the PL score path. With a single positive, the PL/REINFORCE score path is also a high-variance surrogate for a supervised listwise loss (softmax/ListMLE), which is never compared. The gains may therefore come from fine-tuning a cross-encoder-style scorer rather than from "reasoning to rank".
  2. The evaluation protocol favors the proposed method.
    • Negatives are sampled uniformly from the whole catalogue (App. A.3), so most are unrelated to the user. Item text includes popularity cues (rating, number of buyers; Fig. 6), yet there is no popularity baseline and no evaluation on retriever-selected candidates (e.g., SASRec top-20).
    • The conventional baselines are at or near the random level (NDCG@10 ≈ 0.227 for 1 of 20) on Industrial and Movies & TV (Table 2). This is likely because test users and most Industrial positives are unseen in training, which ID-based models cannot handle.
    • The LLM baselines were designed for other tasks: item generation (LatentR3), full-catalogue ranking (R2EC) and binary classification (TALLRec). How they were adapted to score 20 candidates is not described. R2Rank alone is trained directly on this protocol, and its Industrial gain over LatentR3 concentrates on unseen items (+0.287 vs 11.3% relative on warm items; Table 10).
  3. Inference cost and variance are not reported. R2Rank generates one rationale per candidate (K = 20 generations per user, growing linearly with K), but latency and tokens per user are never reported, although the paper emphasizes the industrial setting. All results also appear to be single runs, while shuffling the history order alone gives std 0.085 (Table 11).
Questions for authors
  1. Can you evaluate R2Rank and LatentR3 on retriever-selected candidates (e.g., the top-20 from SASRec) on at least one dataset, and report a popularity baseline (e.g., ranking by number of buyers or training frequency)? Can you also report results restricted to warm items (seen in training) on each dataset?
  2. How were LatentR3 (an item-generation model), R2EC (a full-ranking model) and TALLRec (a binary classifier) adapted to score 20 fixed candidates? Were they trained on the same candidate sets as R2Rank? How does LightGCN obtain embeddings for user-disjoint test users?
  3. What is the inference cost per user (latency, generated tokens, rationale length) compared with the LLM baselines and a single listwise call, and how does it scale with K?
2 comments
AuthorsOpenReview ↗

Responses to Reviewer x8T2 [1/2]

Thank you for your careful review and constructive feedback. Below, we provide additional results and clarifications to address your questions and concerns.

W1. Scoring architecture, CoT, and the ranking objective.

On Industrial, the no-CoT variant reaches 0.758 NDCG@10, compared with 0.560 for LatentR³ and 0.818 for Full. A substantial advantage is therefore present without explicit rationales. The 0.060 difference compares the full and no-CoT configurations; it is not an estimate of PPO's contribution. On Musical, plain CoT also underperforms no CoT (0.494 vs. 0.545), so the results do not support a general claim that generating more reasoning text improves ranking. That template comparison also changes the retained SFT examples and response lengths, so it does not isolate self-checking alone.

Common Response C1 addresses the optimization question on Musical Instruments and Industrial, retaining rationale generation and omitting recommendation-specific SFT in every configuration. Full outperforms PL-only on both datasets, supporting an additional benefit from explicitly optimizing rationale-token probabilities beyond training the scorer through PL. Full also outperforms PPO with head-only PL, supporting the contribution of ranking gradients to backbone adaptation. PL-only can still change generation indirectly through the shared backbone. These comparisons concern the RL-only setting and do not directly decompose the main SFT & RL gains discussed above.

W2 and Q1. Candidate difficulty, popularity, and warm items.

Common Response C2 compares uniform negatives with SASRec-selected K=20 candidates on the complete Musical Instruments test set. R2Rank retains an advantage over LatentR³, but the gap narrows from 0.057 under uniform sampling to 0.022. The result supports an advantage beyond random negatives in this setting, with a smaller margin under retrieval-based candidate selection.

MostPop ranks candidates by their training interaction counts and reaches 0.348 NDCG@10 under uniform sampling and 0.355 with SASRec-selected candidates, below both personalized models in both protocols. Popularity alone therefore falls short of their performance in these comparisons. This does not rule out their use of popularity cues embedded in item text.

For the requested seen-item analysis, C2 restricts evaluation to positives with at least one main-RL training occurrence. R2Rank remains ahead on all four datasets, including Industrial, where it reaches 0.866 versus 0.672 for LatentR³. The advantage therefore extends to seen positives. This subset includes all positive items with at least one training occurrence; the original Industrial “Warm” stratum included only items with more than five. The reported 68.7% zero-frequency fraction remains defined against the main RL-interaction corpus, rather than every source of SFT or pretraining exposure.

W2 and Q2. Baseline adaptation and unseen users.

All trainable baselines and R2Rank share the same main training data and training candidate sets. All methods rank the same fixed candidate sets at evaluation. All trainable LLM baselines use Qwen2.5-3B-Instruct, while retaining their method-specific training objectives. TALLRec scores each user–candidate pair using the positive-response probability normalized over its positive and negative responses. R²EC scores each candidate using its native recommendation head.

LatentR³ first derives latent reasoning from the user-history prompt and then generates an item title. Its official inference implementation uses prefix-constrained beam search and returns length-penalized sequence scores, with a default length penalty of 1.0. In our fixed-candidate evaluation, we retain the latent reasoning and score each candidate title using the corresponding length-normalized conditional log-likelihood:

s(u,i)=1∣Yi∣∑t=1∣Yi∣log⁡Pθ(Yi,t∣xu,ru,Yi,<t),s(u,i)=\frac{1}{|Y_i|}\sum_{t=1}^{|Y_i|}\log P_{\theta}(Y_{i,t}\mid x_u,r_u,Y_{i,<t}),

where xux_u is the user-history prompt, rur_u is the latent reasoning, and YiY_i is the candidate-title response sequence, retaining the original response formatting and end-of-sequence token. We evaluate all 20 candidate responses with teacher forcing and sort their scores in descending order, obtaining a complete ranking without relying on whether beam search retrieves each candidate. This adapts the generative sequence score to the fixed candidate set; the ground-truth item's negative perplexity remains the RL training reward, a separate role from inference-time candidate scoring.

AuthorsOpenReview ↗

Responses to Reviewer x8T2 [2/2]

For user-disjoint test users, LightGCN constructs user representations by aggregating the embeddings of items in each user's observed history. Candidate items with no training interactions receive a score of zero in our implementation, so these candidates share the same fallback score. SASRec and BERT4Rec encode the observed item history to represent a user. User-disjoint evaluation and item cold start therefore concern different sources of information: a new user's visible history can provide preference evidence, while zero-interaction items lack observed collaborative evidence. C2 separately reports results on seen positives to characterize the item-frequency effect.

W3 and Q3. Inference cost.

Common Response C3 reports the existing serving benchmark on one H20 in bf16 with eight concurrent users. R2Rank's P50 latency is 5.80 seconds at K=20 and 12.70 seconds at K=50, compared with 1.92 and 4.35 seconds for LatentR³. Its P95 latency rises from 7.90 to 17.10 seconds. Thus, increasing K from 20 to 50 increases R2Rank's median waiting time by 2.19 times in this benchmark.

W3. Training seeds and significance testing.

The main results and additional ablations each use one training seed. The significance markers in the manuscript use two-sided paired Wilcoxon signed-rank tests on per-test-instance NDCG values, pairing the methods on the same evaluation instance. The thresholds are p<0.05/0.01/0.001p<0.05/0.01/0.001 for †/‡/§, respectively. These are instance-level comparisons for the evaluated checkpoints. Table 11 separately measures sensitivity to history order for a fixed checkpoint: its standard deviation of 0.085 describes performance changes under history permutations.

Reviewer_yLebOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

R2Rank scores each candidate separately. For each (user, candidate) pair, the LLM writes a rationale (reasoning + self-check + decision). A shared MLP reads the last-token hidden state of the rationale and outputs a score. The final list sorts candidates by this score. Other candidates never appear in the prompt.

Training: sample a top-10 list from a Plackett-Luce (PL) distribution over the scores, and use its NDCG@10 as the reward. This reward trains two things: (1) the rationale policy, via token-level PPO; (2) the scores, via the PL log-prob, whose gradient goes through the MLP into the backbone (hidden states are not detached). Before RL, an SFT stage distills rationales from DeepSeek-R1 and keeps only those whose final decision matches the observed label.

Experiments: three Amazon datasets and one industrial ads dataset. Each test list has 1 positive and 19 random negatives (K=20); metrics are NDCG@1/5/10. The 3B model beats all trainable LLM baselines with the same backbone on all four datasets at NDCG@10 (largest gap +0.258 over LatentR3 on Industrial), but loses to zero-shot DeepSeek-R1 (671B) on Movies & TV. Analyses test backbone training, SFT, CoT, and whether rationale content matters (frozen model, swap rationales between candidates).

Critical strengths

1. The score really depends on the generated rationale, and the ranking reward really trains the backbone through it. Table 3 shows joint training beats frozen backbone + MLP. Table 7 shows that, with the model frozen, swapping rationales between candidates drops NDCG@10 (Industrial 0.818 -> 0.682). So the generated content is used by the score, not decoration. This is a real step beyond prior pointwise-RL and latent-token methods.

2. The analyses are honest about where the method fails. Table 10 and Figure 2 show the gain concentrates on cold items, and Figure 2a shows R2Rank is worse than SASRec/BERT4Rec on frequent items. Table 11 shows sensitivity to history order. Few papers report their own losing cases; this makes the claims checkable.

Critical weaknesses

1. The central claim - ranking utility optimized into the reasoning process - is never tested. Only the PPO path changes which tokens are generated, and there is no PPO-off or PL-off ablation; and its advantage is one list-level NDCG shared by all tokens of all 20 candidates, so the gradient cannot tell which candidate's reasoning helped. This coupling is the only new idea over R2EC and LatentR3, so the main contribution remains an assumption.

2. The headline industrial gain is an artifact of the protocol. The task - 1 positive against 19 random negatives, with 68.7% of industrial test positives unseen in training - is per-candidate text matching: the win over ID baselines is modality, and the win over LLM baselines is architecture fit to this easy task, not reasoning. Where the task starts to look like real ranking the method breaks: it loses to SASRec/BERT4Rec on frequent items (Figure 2a) and drops from 0.564 to 0.356 at K=50 (Table 12).

Questions for authors

1. Same backbone, SFT data, candidates and budget: report the full method, PL-only (remove PPO), PPO-only (PL trains only the head), and PPO with a per-candidate advantage. Which one keeps the gains in Tables 2 and 4? If PL-only equals the full method, the reasoning-policy claim should be dropped.

2. The paper removes all other candidates from each assessment. How is it established that this isolation loses no information compared to letting candidates see each other? Independent scoring cannot express relative judgment and the listwise objective couples candidates only through the shared scoring head. What evidence shows that candidate-independent reasoning is sufficient, and in what cases would the presence of other candidates change a candidate's correct score?

2 comments
AuthorsOpenReview ↗

Responses to Reviewer yLeb [1/2]

Thank you for your careful review and constructive feedback. Below, we provide additional results and clarifications to address your questions and concerns.

W1 and Q1. Which gains remain without token PPO? The rationale-substitution experiment in Table 7 tests whether the scorer uses rationale content; it does not by itself establish that explicitly optimizing rationale generation improves ranking.

Common Response C1 compares Full (PL + PPO), PL-only, and PPO with head-only PL on Musical Instruments and Industrial, with MLP-only as a frozen-backbone reference. All variants use Qwen2.5-3B-Instruct without recommendation SFT. PL-only retains rationale generation and allows the PL loss to update the backbone through the scoring head. Your “PPO-only” variant retains PL training for the head but stops the PL gradient at the head input; token PPO still updates the backbone.

The key quantity is how much of the advantage in this RL-only setting survives without token PPO. On Industrial, PL-only achieves 0.575, compared with 0.709 for Full and 0.560 for LatentR³. Its remaining absolute gap over LatentR³ is 0.015, compared with 0.149 for Full; removing token PPO lowers NDCG@10 by 0.134. On Musical Instruments, PL-only reaches 0.476 versus 0.541 for Full, a difference of 0.065. PPO with head-only PL reaches 0.503 on Musical and 0.628 on Industrial, respectively 0.038 and 0.081 below Full. These single-seed point estimates support an incremental benefit from each update path when the other is retained. The comparisons concern training without recommendation SFT and do not directly decompose the main Industrial result of 0.818 after SFT and RL, or its 0.258 gap over LatentR³.

One distinction matters for interpreting PL-only: because PL updates the shared backbone, it can also change subsequent rationale generation indirectly. Token PPO is the explicit term that optimizes the probabilities of the sampled rationale tokens. The observed Full–PL-only differences therefore support the additional value of this explicit update within the tested training procedure; they do not assign all of the gain to PPO or establish the faithfulness of the generated reasoning.

Shared return and candidate-level credit. We use the sampled list-level NDCG as the return, with a zero baseline. Each candidate contributes the log-probability gradient of its own generated tokens, weighted by that return. Across rollouts, these candidate-specific gradients can correlate differently with ranking quality, supplying a learning signal for the joint rationale policy. Before PPO clipping, this token-policy term follows the joint expected ranking utility in Equation 6; Equation 7 supplies the clipped surrogate used for the update. A single rollout does not identify each rationale's individual contribution and can have high variance. The C1 controls keep the reward definition fixed and show the contribution of each update path under this training procedure.

W2. Is the Industrial gain explained by the evaluation protocol? We agree that the frequency distribution and random negatives are central to interpreting this result. The 68.7% figure denotes test positives with zero occurrences in the main RL training interactions. User-disjoint auxiliary SFT sampling does not itself establish item-disjointness; this figure therefore does not establish absence from all training data.

Common Response C2 restricts evaluation to positives with at least one main-RL training occurrence. R2Rank retains an advantage over LatentR³ on all four datasets, reaching 0.866 versus 0.672 on Industrial. This subset includes both the 1–5 and >5 occurrence groups; the original “Warm” subset and its 11.3% relative gain refer only to the latter. The seen-positive results show that the advantage is not confined to zero-frequency items, without establishing a causal explanation for the gap.

Candidate difficulty is a separate issue. In C2, both protocols use the full Musical Instruments test set. R2Rank retains an advantage over LatentR³ with SASRec-selected K=20 candidates, although the NDCG@10 margin is smaller: 0.022, compared with 0.057 under the original Random-20 protocol. This extends the evidence to a retrieved candidate pool on one public dataset; it does not settle how much candidate difficulty explains the Industrial gain.

We retain the unfavorable frequent-item comparison in Figure 2a. For the larger candidate sets, Table 12 already provides a matched baseline comparison: R2Rank decreases from 0.564 to 0.356, while LatentR³ decreases from 0.507 to 0.298. The absolute gap is therefore 0.057 at K=20 and 0.058 at K=50. R2Rank’s absolute performance declines, but its margin over LatentR³ is maintained in this experiment. This evidence is limited to two random-negative candidate sizes on Musical Instruments and does not substitute for the retrieval-based comparison.

AuthorsOpenReview ↗

Responses to Reviewer yLeb [2/2]

Q2. When would access to other candidates change the judgment? Independent scoring can express relative preference through score differences, and a candidate’s rank depends on its competitors. If R2Rank prefers A to B, adding a stronger candidate C can produce the order C, A, B. What it cannot express is a reversal from A, B to B, A caused by the presence of C. For example, if C is nearly identical to A, selecting C may make B the more valuable addition to a diverse slate. Independent assessment removes candidate presentation order from each judgment, but the current experiments do not establish that this isolation preserves all information relevant to ranking. Our evidence concerns individual relevance under the evaluated protocol; slate-dependent objectives involving diversity, redundancy, or complementarity may require joint candidate modeling.

Reviewer_vkPUOpenReview ↗
Rating
2
Confidence
3
Informativeness
3(+1)
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes an end-to-end framework, Reasoning to Rank (R2Rank), designed to align LLM reasoning with listwise ranking utility. To mitigate candidate-position bias, R2Rank decouples the recommendation process into independent pointwise inferences, where each candidate item generates a structured rationale mapped to a relevance score via an MLP head. It then leverages a Plackett-Luce (PL) probabilistic surrogate combined with Proximal Policy Optimization (PPO) to train the LLM end-to-end using a shared Monte Carlo NDCG return. The framework also incorporates an initial supervised fine-tuning stage using teacher-generated reasoning traces with a self-check mechanism. Experiments on three Amazon datasets and one industrial dataset demonstrate improvements over conventional and LLM-based recommendation baselines.

Critical strengths
  1. The idea of connecting candidate-specific reasoning to listwise ranking utility is interesting and is demonstrated to be effective for listwise recommendation, particularly in scenarios with sparse interaction histories.
  2. This paper presents a simple yet effective solution for mitigating candidate-presentation-order bias in LLM-based recommendation through independent user–item assessments.
  3. Extensive experiments are conducted to show competitive performance against conventional and LLM-based recommendation methods, with particularly substantial improvements on the industrial dataset.
Critical weaknesses
  1. Although the paper discusses relevant research on LLM-based recommendation, the methodological differences between the proposed approach and existing reasoning-based recommendation frameworks, particularly R²EC, require more detailed analysis.
  2. The proposed framework does not explicitly model interactions among candidate items, which may limit its ability to capture relative preferences and inter-item dependencies. Further investigation is needed to determine whether independent candidate-level reasoning is sufficient for more complex recommendation scenarios.
  3. In the ablation study, the paper does not isolate the individual contributions of the Plackett-Luce ranking objective versus the PPO-based reasoning optimization, leaving the necessity of the joint mechanism underexplored.
  4. R2Rank independently generates reasoning traces and computes relevance scores for each candidate, introducing considerable inference overhead compared with conventional recommendation models. A systematic comparison of inference latency, token consumption, and GPU-hours across different training and inference stages would help assess the efficiency of the proposed approach.
Questions for authors
  1. The proposed framework jointly optimizes reasoning generation and relevance scoring through PPO and the Plackett–Luce objective. Could the authors provide more details on how these two optimization paths interact and contribute to the learned recommendation representations?
  2. During training, are any normalization or weighting strategies used to prevent one optimization path from dominating the updates to the LLM backbone?
  3. The PPO objective aggregates token-level contributions from all candidate-specific reasoning trajectories. How does the implementation handle differences in reasoning length across candidates? Are the token-level objectives normalized to avoid introducing a preference for longer reasoning trajectories?
2 comments
AuthorsOpenReview ↗

Responses to Reviewer vkPU [1/2]

Thank you for your careful review and constructive feedback. Below, we provide additional results and clarifications to address your questions and concerns.

W1. Distinction from existing reasoning-based recommenders. R²EC and R2Rank both combine explicit reasoning with a recommendation head and RL-style training. The distinction we emphasize concerns the conditioning of the reasoning and its coupling to ranking utility. R²EC generates a reasoning trace from the user’s history and uses the resulting user representation to score multiple items. In R2Rank, each rationale is conditioned on both the user context and the candidate, yk∼πθ(⋅∣cu,xk)y_k\sim\pi_\theta(\cdot\mid c_u,x_k). This allows different candidates to draw on different evidence from the same history. LatentR³ instead uses latent reasoning tokens for next-item generation.

During training, a shared MLP maps the candidate-specific representations to scores that jointly define a PL distribution over ranked prefixes. The NDCG@10 of a sampled prefix provides a common return: the PL term updates the scoring head and backbone representations, while token PPO updates the probabilities of the generated rationale tokens. At inference, each candidate is assessed independently and candidates are sorted by their scores. The coupling therefore occurs through listwise training utility rather than through a joint candidate-list prompt.

Our contribution is to train candidate-conditioned reasoning for cutoff-level ranking utility. We build on SFT to initialize the structured reasoning format and on PL and PPO to optimize the resulting ranking and rationale policy jointly.

W2. Relative preferences and candidate interactions. Although each rationale is generated independently, the shared MLP is trained through a distribution that compares scores across the list. For two candidates A and B, the ratio of their PL selection probabilities at any step where both remain available is exp⁡(sA−sB)\exp(s_A-s_B). Their score difference therefore expresses a relative preference, and this preference can differ across users because both scores depend on the user’s history. Table 3 illustrates the ranking signal available to the head: even with the backbone frozen, the trained MLP achieves 0.528 NDCG@10 on Industrial in the RL-only setting. This result supports learning useful ranking scores from fixed representations; it does not establish that candidate interactions are unnecessary.

The design distinguishes presentation order from list composition. Moving an item to a different input position should not change its assessment, yet our position-sensitivity experiment shows that listwise prompting can exhibit this behavior. Independent user–item prompts remove candidate presentation order from each assessment. They do not model cases where adding a third item should reverse the preference between A and B. For example, selecting an item similar to A may make B more valuable in a slate that rewards diversity or penalizes redundancy. Our evidence supports the evaluated relevance-ranking setting; it does not establish that independent scoring captures these slate-dependent preferences.

W3 and Q1. Roles of PL and PPO. The two paths use the same return but differentiate different probabilities. Given the generated rationales, the PL term updates the probability of the sampled ranking through the scoring head and the backbone representations it consumes. Token PPO directly updates the probabilities of the sampled rationale tokens. Both paths update the shared backbone, while the scoring head receives the PL term. Because the backbone is shared, PL updates can also change subsequent generation indirectly; removing token PPO does not freeze the reasoning model.

Common Response C1 separates these paths on Musical Instruments and Industrial, with rationale generation retained and no recommendation-specific SFT in any variant. Full achieves the highest NDCG@10 on both datasets. Its comparison with PL-only supports an additional benefit from explicitly optimizing rationale-token probabilities; the comparison with PPO + head-only PL supports allowing the ranking gradient to adapt backbone representations. These controls complement the frozen-backbone MLP-only comparison in Table 3.

AuthorsOpenReview ↗

Responses to Reviewer vkPU [2/2]

Q2–Q3. Loss weighting and rationale length. Algorithm 1 specifies JPPO+JPL\mathcal J_{\mathrm{PPO}}+\mathcal J_{\mathrm{PL}} for the backbone, while the scoring head receives only JPL\mathcal J_{\mathrm{PL}}. Both terms have unit coefficients in this objective. The additive form is motivated by Equation 6: the token-policy and score-path contributions arise from differentiating the same joint expected ranking utility. Equation 7 replaces the token-policy contribution with a clipped PPO surrogate. These fixed coefficients reflect the decomposition; the two paths can still have different gradient magnitudes.

Regarding rationale length, Equation 7 uses the shared NDCG return with a zero baseline and sums token contributions without dividing by each rationale’s length. Token summation is motivated by the sequence log-probability decomposition,

log⁡πθ(yk∣cu,xk)=∑t=1∣yk∣log⁡πθ(ok,t∣cu,xk,ok,<t)\log \pi_\theta(y_k\mid c_u,x_k)=\sum_{t=1}^{|y_k|}\log \pi_\theta(o_{k,t}\mid c_u,x_k,o_{k,\lt t})

with PPO providing a practical surrogate for the token-policy update. Longer rationales contain more objective terms, but this alone implies neither proportionally larger gradient magnitudes nor a learned preference for longer outputs. Dividing each rationale’s contribution by its length would change the relative weighting of trajectories.

W4. Computational cost. The existing serving benchmark reports P50/P95 latency of 5.80/7.90 seconds per user at K=20K=20, rising to 12.70/17.10 seconds at K=50K=50, on one H20 in bf16 with eight concurrent users. The measurements include tokenization, generation, scoring, and sorting. Common Response C3 provides the latency and throughput comparison: R2Rank's median latency is approximately three times that of LatentR³. For training, the 3B model takes roughly 40 elapsed hours per RL epoch on eight H20 GPUs, corresponding to approximately 320 allocated H20 GPU-hours per RL epoch. These measurements make the additional cost explicit; candidate independence does not remove that overhead.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Common Responses to all Reviewers [1/2]

We thank the Area Chair and all three reviewers for their constructive feedback. Below, we provide additional experiments and clarifications to address the questions and concerns raised in the reviews.

C1. Contributions of PL and PPO.

We compare the two optimization paths on Musical Instruments and Industrial, using Qwen2.5-3B-Instruct, K=20, and no recommendation-specific SFT. Every configuration retains rationale generation. PL-only removes token PPO while allowing PL to update both the backbone and scoring head. PPO + head-only PL retains token PPO but blocks PL gradients into the backbone, corresponding to reviewer yLeb's “PPO-only” control.

Results are single-seed NDCG@10 values. Full uses both update paths without SFT; its scores and those of MLP-only reproduce the RL-only results in Table 3.

Configuration (all without SFT)MusicalIndustrial
Full: PL + PPO (Table 3, LLM+MLP)0.5410.709
PL-only0.4760.575
PPO + head-only PL0.5030.628
MLP-only (Table 3, frozen backbone)0.3680.528

Full performs best on both datasets. The PL-only comparison supports an additional benefit from explicitly optimizing rationale-token probabilities, while the head-only PL comparison supports allowing ranking gradients to adapt the backbone. Together, these results support retaining both update paths in the evaluated RL-only setting.

C2. Candidate difficulty and cold-start effects.

We separate the frequency of the positive item from the way competing candidates are selected. The existing Industrial breakdown uses the positive item's frequency in the main RL-interaction training corpus:

Positive-item frequencyShare of test positivesR2Rank NDCG@10LatentR³ NDCG@10Absolute difference
0 occurrence68.7%0.7960.509+0.287
1–5 occurrences21.6%0.8520.611+0.241
More than 5 occurrences9.7%0.8980.807+0.091

The 0.091 difference in the last row is an 11.3% relative improvement over LatentR³. This shows that the reported advantage is not confined to positives absent from the main RL-interaction corpus. It is a descriptive comparison across subsets, rather than an isolated estimate of the causal effect of cold start. Moreover, zero frequency in this corpus does not establish absence from every source used in SFT or pretraining.

For the warm-only analysis across all four datasets, we define seen positives as items with at least one occurrence in the main RL-interaction training corpus. We restrict the original Random-20 evaluation to these positives while retaining their candidate sets. This threshold differs from the Industrial “Warm” row above, which uses more than five occurrences.

DatasetR2Rank NDCG@10LatentR³ NDCG@10Absolute difference
Musical0.6050.548+0.057
Movies & TV0.5910.521+0.070
Video Games0.6220.554+0.068
Industrial0.8660.672+0.194

R2Rank remains ahead on seen positives across all four datasets, indicating that its advantage is not confined to items absent from the training interactions.

Seen-item results do not resolve candidate difficulty. We therefore evaluate SASRec-selected candidates on Musical Instruments. Both candidate protocols cover the complete test set. R2Rank and LatentR³ use their main-table checkpoints and rank identical candidate sets of K=20 within each protocol.

We also evaluate MostPop, which ranks candidates by their interaction counts in the training corpus. All methods rank the same candidates within each protocol.

Candidate protocol on MusicalKR2Rank NDCG@10LatentR³ NDCG@10MostPop NDCG@10R2Rank − LatentR³
Uniform negatives200.5640.5070.348+0.057
SASRec-selected candidates200.5090.4870.355+0.022

R2Rank remains ahead on SASRec-selected candidates, although its margin is smaller than with uniform negatives. MostPop trails both personalized models under both protocols, showing that ranking solely by training-set popularity does not recover their performance.

AuthorsOpenReview ↗

Common Responses to all Reviewers [2/2]

C3. Computational cost.

R2Rank generates one rationale per candidate. We report the elapsed time for ranking a user's complete candidate set and the throughput under concurrent requests.

The existing serving benchmark uses one NVIDIA H20 in bf16 with eight concurrent users. Timing covers tokenization, generation, scoring, and sorting. P50 and P95 are per-user latency quantiles under that concurrency; throughput is reported separately.

KMethodP50 latency/userP95 latency/userUsers/sCandidates/s
20R2Rank5.80 s7.90 s1.2024.0
20LatentR³1.92 s2.74 s3.3567.0
20SASRec3.1 ms5.4 ms1,46029,200
50R2Rank12.70 s17.10 s0.5628.0
50LatentR³4.35 s6.10 s1.4874.0
50SASRec4.8 ms7.3 ms1,06053,000

R2Rank's median latency is approximately three times that of LatentR³, and both are substantially slower than SASRec. Increasing K from 20 to 50 raises R2Rank's median latency by 2.19 times. These figures quantify the serving cost of candidate-level reasoning.

For training, the 3B RL stage takes approximately 40 elapsed hours per epoch on a node with eight H20 GPUs, corresponding to approximately 320 allocated H20 GPU-hours per RL epoch.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.