- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper asks whether an LLM can rerank a supplied candidate slate and write a rationale when every history item and candidate appears only as a Semantic ID (SID), with no item text. It builds a frozen, user-disjoint testbed (RERANKID) on Kuaishou short-video data and the OpenOneRec 1.7B and 8B checkpoints. The testbed renders the same users and slates as SIDs, captions, or both, and varies history length and slate size one axis at a time. The released checkpoints mostly fail at the listwise interface, often returning the input order or looping. Four post-training signals are then compared. The headline result is an 8B RLVR policy with a graded NDCG@5 reward that reaches 0.277 NDCG@5 from SID-only prompts, beats Random with no history, keeps improving with longer histories, and matches text-reading frontier LLMs whose prompts are about 20 times longer. An LLM-judged rationale audit finds that this policy argues trade-offs well but frequently misdescribes specific items.
Critical strengths
- Carefully controlled design. Item-matched renderings make metadata access a controlled variable on identical users and slates. Nested one-axis interventions, user-disjoint splits, paired bootstrap CIs, and zero-scored invalid outputs with separate interface diagnostics make the comparisons clean.
- Transparent reporting. Null and negative results are reported plainly: RFT trails SFT at 8B, and RLVR gives no gain at 1.7B. Parity with text references is called “on par” because the CIs include zero. The extraction fallback is fully accounted for, and the rationale audit exposes the method’s own hallucinations.
- Interesting, practical findings. The SID policy matches text readers with prompts about 20× shorter and stays robust at long histories. The dissociation between ranking and description motivates a concrete design: rank from SIDs, then explain the top items using metadata.
Critical weaknesses
-
The “item-level preference signal” may be popularity or exposure. Each pool contains the positive, a mean of 6.6 exposure-hard negatives, and uniform-random catalog fill. At N=30, roughly 22 of 30 candidates are therefore random items the platform may never have shown. SID embeddings co-trained on behavior sequences plausibly encode popularity or exposure likelihood, which captions do not reveal. That alone would explain both the margin over Random (+0.079) where Gemini is at chance, and the parity with text readers at the anchor. A popularity or exposure-frequency baseline, plus results restricted to the positive vs. exposure-hard negatives, are needed to separate personalized preference from separating exposed items from random fill. Without them, the claim holds only in a much narrower form.
-
Missing likelihood-based baseline from the same backbone. The conclusion that Base models cannot rank rests only on greedy free-form generation, in a format the checkpoints never trained on. Their native skill is next-item SID generation, so the natural control is to score each candidate by the Base model’s log-likelihood of its SID given the history (since one can use parallel decoding, it brings not much computational burden). If this approaches RLVR, post-training mainly teaches an output format and reasoning adds little to ranking. If it falls well short, the paper’s case is stronger.
-
The rationale-audit asymmetry partly reflects the SFT template, not the representation. The teacher prompt (Appendix C.1) requires discussing one or two alternatives placed exactly second and third, consistent with the final ranking, and forbids enumerating candidates. RLVR inherits this from SFT, while Gemini answers the generic evaluation prompt. The 50:0 trade-off verdict and the low coverage are what this template produces. The item-claim and user-portrait findings are less affected. Giving Gemini the same template, or using the teacher’s own traces as the reference, would make the comparison fair.
Questions for authors
Please refer to the weaknesses.
3 comments
W2. Missing likelihood-based baseline from the same backbone. The conclusion that Base models cannot rank rests only on greedy free-form generation, in a format the checkpoints never trained on. Their native skill is next-item SID generation, so the natural control is to score each candidate by the Base model’s log-likelihood of its SID given the history (since one can use parallel decoding, it brings not much computational burden). If this approaches RLVR, post-training mainly teaches an output format and reasoning adds little to ranking. If it falls well short, the paper’s case is stronger.
Response. We added this control (new rows in Table 1, new Appendix D.6). Each candidate is scored by the Base model's log-likelihood of its SID tokens given the history, in the next-item format of pretraining.
| System | NDCG@5, k=0 | NDCG@5, k=64 |
|---|---|---|
| SID likelihood, 8B Base | .330 | .415 |
| 8B Base, reason then rank | .140 | .168 |
| 8B RLVR, reason then rank | .184 | .277 |
| GPT (text) | n/a | .256 |
The likelihood ranking is .138 above 8B RLVR (CI +.097 to +.179) and above every text reader. It is the central evidence for the paper's position, in both directions.
- Potential: the backbone holds the ranking signal that motivates SID-only reranking, from a few hundred SID tokens.
- Not yet sufficient: the Base models fail at the reason-then-rank interface (Section 4), and post-training teaches the interface but carries over only part of this signal. The gap to the likelihood ranking is the target the paper sets for future post-training.
Reasoning is not expected to add accuracy, and it does not: generating the list without the rationale changes NDCG@5 by at most .012 (new Appendix D.7). Appendix A.4 notes that pretraining included the evaluation users, so the likelihood row measures what the backbone holds and is not a deployable baseline.
W3. The rationale-audit asymmetry partly reflects the SFT template, not the representation. The teacher prompt (Appendix C.1) requires discussing one or two alternatives placed exactly second and third, consistent with the final ranking, and forbids enumerating candidates. RLVR inherits this from SFT, while Gemini answers the generic evaluation prompt. The 50:0 trade-off verdict and the low coverage are what this template produces. The item-claim and user-portrait findings are less affected. Giving Gemini the same template, or using the teacher’s own traces as the reference, would make the comparison fair.
Response. We agree and ran the matched comparison (new Appendix D.8). Gemini wrote its rationales under the teacher's requirements, with no positive revealed, and the paired audit was repeated against the unchanged RLVR rationales.
| Aspect | Gemini, generic prompt : RLVR | Gemini, teacher template : RLVR |
|---|---|---|
| Trade-off argument | 0 : 50 | 43 : 2 |
| Coverage | 100 : 0 | 6 : 1 (93 ties) |
| Item-claim accuracy | 66 : 0 | 79 : 0 |
| User portrait | 37 : 0 | 19 : 1 (69 ties) |
As the reviewer expected, the trade-off verdict and the coverage gap follow the template, and the item-claim result does not. Section 5.2 is updated accordingly. A new statement-level check gives absolute rates: 63% of what RLVR states about items is supported by the item caption and 19% is contradicted, against 96% and 0.5% for Gemini under the same template. A rationale that is faithful to the items is the second open problem the paper poses.
Thank the authors for the detailed response, which has largely addressed my concerns. I will keep my positive score.
We thank the reviewer for the careful reading. All three requested controls are now in the paper. Before the individual points, we state the purpose of the paper, which the revision now makes explicit in the abstract and introduction. All changes are marked in blue.
W1. The “item-level preference signal” may be popularity or exposure. Each pool contains the positive, a mean of 6.6 exposure-hard negatives, and uniform-random catalog fill. At N=30, roughly 22 of 30 candidates are therefore random items the platform may never have shown. SID embeddings co-trained on behavior sequences plausibly encode popularity or exposure likelihood, which captions do not reveal. That alone would explain both the k=0 margin over Random (+0.079) where Gemini is at chance, and the parity with text readers at the anchor. A popularity or exposure-frequency baseline, plus results restricted to the positive vs. exposure-hard negatives, are needed to separate personalized preference from separating exposed items from random fill. Without them, the claim holds only in a much narrower form.
Response. We agree and added both controls (new Table 3, new Appendix D.6): a popularity baseline, which counts how often each item occurs in other users' histories, and every result split by negative type. The last two columns give the probability that the positive is ranked above a random-fill negative and above an exposure-hard negative (chance is .500).
The primary metric still separates systems: a ranker that solved coarse relevance and guessed among exposed items would score .416 NDCG@5, and every reasoned ranking is at .28 or below. These controls serve what the paper is for: it evaluates current methods on a new task and shows where they fall short, and it does not present 8B RLVR as a strong reranker.