acceptodds
Under review as a conference paper at ICLR 2027

Reasoning Reranking with Semantic IDs as the Sole Item Representation

Abstract

Reasoning rerankers write a rationale before ranking a candidate slate, and existing systems read item metadata to do so, even though generative recommenders increasingly represent items as compact Semantic IDs (SIDs). Ranking from SIDs alone has two appeals: SID prompts are far shorter than the matched item text, and although item codes are fixed from content, the backbone learns their token embeddings in part from user-item interaction sequences, so the tokens can carry behavioral signal that item text lacks. That training targets next-item generation, and whether it supports ranking a supplied slate from SIDs alone is untested. We build ReRankID, a controlled testbed in which models reason and rank from a history and slate shown only as SIDs, varying item rendering, history length, and slate size one at a time. On this testbed, such models often copy the input order instead of reranking, and readable metadata does not repair the failure. We compare supervised fine-tuning, rejection-sampled self-training, on-policy distillation, and verifier-based reinforcement learning as post-training signals and find that they elicit the missing behavior. The strongest policy, an 8B model trained with a graded ranking reward, gives the best SID-only ranking and matches strong text-reading LLMs whose prompts are about 20x longer. It ranks above random with no history at all, where a caption-reading reference does not, and it keeps its gains at long histories, where text prompts pass 50k tokens and the text reference stops improving. A paired rationale audit separates ranking from description: the policy argues its trade-offs more sharply than the text reference, but its claims about specific items are often wrong. SIDs are enough to learn the ranking; describing the items from SIDs alone is an open grounding problem.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Recommender Systems, Semantic IDs, Reasoning, Post Training

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_5g4gOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether an LLM can rerank a supplied candidate slate and write a rationale when every history item and candidate appears only as a Semantic ID (SID), with no item text. It builds a frozen, user-disjoint testbed (RERANKID) on Kuaishou short-video data and the OpenOneRec 1.7B and 8B checkpoints. The testbed renders the same users and slates as SIDs, captions, or both, and varies history length and slate size one axis at a time. The released checkpoints mostly fail at the listwise interface, often returning the input order or looping. Four post-training signals are then compared. The headline result is an 8B RLVR policy with a graded NDCG@5 reward that reaches 0.277 NDCG@5 from SID-only prompts, beats Random with no history, keeps improving with longer histories, and matches text-reading frontier LLMs whose prompts are about 20 times longer. An LLM-judged rationale audit finds that this policy argues trade-offs well but frequently misdescribes specific items.

Critical strengths
  1. Carefully controlled design. Item-matched renderings make metadata access a controlled variable on identical users and slates. Nested one-axis interventions, user-disjoint splits, paired bootstrap CIs, and zero-scored invalid outputs with separate interface diagnostics make the comparisons clean.
  2. Transparent reporting. Null and negative results are reported plainly: RFT trails SFT at 8B, and RLVR gives no gain at 1.7B. Parity with text references is called “on par” because the CIs include zero. The extraction fallback is fully accounted for, and the rationale audit exposes the method’s own hallucinations.
  3. Interesting, practical findings. The SID policy matches text readers with prompts about 20× shorter and stays robust at long histories. The dissociation between ranking and description motivates a concrete design: rank from SIDs, then explain the top items using metadata.
Critical weaknesses
  1. The “item-level preference signal” may be popularity or exposure. Each pool contains the positive, a mean of 6.6 exposure-hard negatives, and uniform-random catalog fill. At N=30, roughly 22 of 30 candidates are therefore random items the platform may never have shown. SID embeddings co-trained on behavior sequences plausibly encode popularity or exposure likelihood, which captions do not reveal. That alone would explain both the k=0k=0 margin over Random (+0.079) where Gemini is at chance, and the parity with text readers at the anchor. A popularity or exposure-frequency baseline, plus results restricted to the positive vs. exposure-hard negatives, are needed to separate personalized preference from separating exposed items from random fill. Without them, the claim holds only in a much narrower form.

  2. Missing likelihood-based baseline from the same backbone. The conclusion that Base models cannot rank rests only on greedy free-form generation, in a format the checkpoints never trained on. Their native skill is next-item SID generation, so the natural control is to score each candidate by the Base model’s log-likelihood of its SID given the history (since one can use parallel decoding, it brings not much computational burden). If this approaches RLVR, post-training mainly teaches an output format and reasoning adds little to ranking. If it falls well short, the paper’s case is stronger.

  3. The rationale-audit asymmetry partly reflects the SFT template, not the representation. The teacher prompt (Appendix C.1) requires discussing one or two alternatives placed exactly second and third, consistent with the final ranking, and forbids enumerating candidates. RLVR inherits this from SFT, while Gemini answers the generic evaluation prompt. The 50:0 trade-off verdict and the low coverage are what this template produces. The item-claim and user-portrait findings are less affected. Giving Gemini the same template, or using the teacher’s own traces as the reference, would make the comparison fair.

Questions for authors

Please refer to the weaknesses.

3 comments

We thank the reviewer for the careful reading. All three requested controls are now in the paper. Before the individual points, we state the purpose of the paper, which the revision now makes explicit in the abstract and introduction. All changes are marked in blue.

  • Why reason: the rationale is the product of a reasoning reranker. Unlike reasoning in mathematics or coding, it is not a means to higher accuracy. At the final ranking stage it lets engineers inspect why a slate was ordered as it was, and it can be shown to users.
  • What the paper contributes: it defines reasoning reranking from SIDs alone as a task, evaluates existing models and common post-training methods on it, and exposes where they fall short. It does not propose a strong model. Its finding is that no current method is sufficient yet, while the pretrained backbone already holds the ranking signal, and it calls for post-training that closes this gap.

W1. The “item-level preference signal” may be popularity or exposure. Each pool contains the positive, a mean of 6.6 exposure-hard negatives, and uniform-random catalog fill. At N=30, roughly 22 of 30 candidates are therefore random items the platform may never have shown. SID embeddings co-trained on behavior sequences plausibly encode popularity or exposure likelihood, which captions do not reveal. That alone would explain both the k=0k=0 margin over Random (+0.079) where Gemini is at chance, and the parity with text readers at the anchor. A popularity or exposure-frequency baseline, plus results restricted to the positive vs. exposure-hard negatives, are needed to separate personalized preference from separating exposed items from random fill. Without them, the claim holds only in a much narrower form.

Response. We agree and added both controls (new Table 3, new Appendix D.6): a popularity baseline, which counts how often each item occurs in other users' histories, and every result split by negative type. The last two columns give the probability that the positive is ranked above a random-fill negative and above an exposure-hard negative (chance is .500).

SystemHistoryNDCG@5vs. random fillvs. exposure-hard
Popularitynone.298.739.500
8B RLVRk=0.184.619.504
Gemini (text)k=0.084.503.492
8B Basek=64.168.588.515
8B RLVRk=64.277.727.542
GPT (text)k=64.256.695.545
  • No history: the margin is an item-level prior of the popularity or exposure kind. It appears against random fill (.619) and not against exposure-hard negatives (.504), and the revision says so.
  • With history: the accuracy of 8B RLVR against random fill rises to .727 (+.109, CI +.075 to +.143). Popularity is constant in k, so this part is personalized.
  • Exposure-hard negatives: no system exceeds .55, text readers included, so fine-grained preference is open for every system. We report it as a diagnostic, since a user has only 6.6 such negatives on average.

The primary metric still separates systems: a ranker that solved coarse relevance and guessed among exposed items would score .416 NDCG@5, and every reasoned ranking is at .28 or below. These controls serve what the paper is for: it evaluates current methods on a new task and shows where they fall short, and it does not present 8B RLVR as a strong reranker.

W2. Missing likelihood-based baseline from the same backbone. The conclusion that Base models cannot rank rests only on greedy free-form generation, in a format the checkpoints never trained on. Their native skill is next-item SID generation, so the natural control is to score each candidate by the Base model’s log-likelihood of its SID given the history (since one can use parallel decoding, it brings not much computational burden). If this approaches RLVR, post-training mainly teaches an output format and reasoning adds little to ranking. If it falls well short, the paper’s case is stronger.

Response. We added this control (new rows in Table 1, new Appendix D.6). Each candidate is scored by the Base model's log-likelihood of its SID tokens given the history, in the next-item format of pretraining.

SystemNDCG@5, k=0NDCG@5, k=64
SID likelihood, 8B Base.330.415
8B Base, reason then rank.140.168
8B RLVR, reason then rank.184.277
GPT (text)n/a.256

The likelihood ranking is .138 above 8B RLVR (CI +.097 to +.179) and above every text reader. It is the central evidence for the paper's position, in both directions.

  • Potential: the backbone holds the ranking signal that motivates SID-only reranking, from a few hundred SID tokens.
  • Not yet sufficient: the Base models fail at the reason-then-rank interface (Section 4), and post-training teaches the interface but carries over only part of this signal. The gap to the likelihood ranking is the target the paper sets for future post-training.

Reasoning is not expected to add accuracy, and it does not: generating the list without the rationale changes NDCG@5 by at most .012 (new Appendix D.7). Appendix A.4 notes that pretraining included the evaluation users, so the likelihood row measures what the backbone holds and is not a deployable baseline.

W3. The rationale-audit asymmetry partly reflects the SFT template, not the representation. The teacher prompt (Appendix C.1) requires discussing one or two alternatives placed exactly second and third, consistent with the final ranking, and forbids enumerating candidates. RLVR inherits this from SFT, while Gemini answers the generic evaluation prompt. The 50:0 trade-off verdict and the low coverage are what this template produces. The item-claim and user-portrait findings are less affected. Giving Gemini the same template, or using the teacher’s own traces as the reference, would make the comparison fair.

Response. We agree and ran the matched comparison (new Appendix D.8). Gemini wrote its rationales under the teacher's requirements, with no positive revealed, and the paired audit was repeated against the unchanged RLVR rationales.

AspectGemini, generic prompt : RLVRGemini, teacher template : RLVR
Trade-off argument0 : 5043 : 2
Coverage100 : 06 : 1 (93 ties)
Item-claim accuracy66 : 079 : 0
User portrait37 : 019 : 1 (69 ties)

As the reviewer expected, the trade-off verdict and the coverage gap follow the template, and the item-claim result does not. Section 5.2 is updated accordingly. A new statement-level check gives absolute rates: 63% of what RLVR states about items is supported by the item caption and 19% is contradicted, against 96% and 0.5% for Gemini under the same template. A rationale that is faithful to the items is the second open problem the paper poses.

Thank the authors for the detailed response, which has largely addressed my concerns. I will keep my positive score.

Reviewer_peGyOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies a new question: can a reasoning reranker use semantic IDs (SIDs) as the sole item representation — with no item text or metadata available at inference — to first write a rationale and then perform listwise reranking over an externally supplied candidate slate? The motivation is two properties of SIDs: the prompt is about 20× shorter than the matched text rendering (nearly 28× at the longest history), and SID token embeddings are learned in part from user–item interaction sequences, so they may carry behavioral/collaborative signal that the item text does not contain.

The authors build ReRankID, a frozen testbed based on Kuaishou short-video data and OpenOneRec (1.7B/8B). It provides three item-matched renderings (SID / text / both) so that "metadata access" becomes a controlled variable, and it evaluates single-positive NDCG@5 while recording interface-failure rates across history length k and slate size N. Under the same protocol, the paper then evaluates four post-training schemes: SFT, RFT, OPD, and RLVR.

Critical strengths
  1. The testbed's engineering rigor and experimental transparency are commendable, but they are methodological and, on their own, not sufficient to support acceptance. The authors isolate "pure-SID reason-then-rerank" into a reproducible task, use three item-matched renderings (SID / text / both) to turn "metadata access" into a controlled variable, and run a rich set of experiments. Overall it is a solid measurement study.
Critical weaknesses
  1. The paper's central claim — that "SIDs carry behavioral signal that the caption text cannot provide" — is not actually established by the experiments. Because the SID representations are taken directly from the upstream pretrained version and kept frozen throughout, the authors never manipulate them as a variable in a controlled comparison; combined with the use of a single model (OpenOneRec) and a single dataset (Kuaishou), it is impossible to separate whether the gains come from a SID-specific signal or from priors already baked into that model's pretraining.

  2. The title foregrounds "Reasoning Reranking," yet the paper provides no evidence that the reasoning actually helps the ranking. During training the reward looks only at whether the final ranking is correct and not at the reasoning text at all (Section 5.1; Appendix C.4), and the authors run no "with-reasoning vs. without-reasoning" comparison. There is therefore no way to tell whether the improved ranking is attributable to the reasoning.

Questions for authors
  1. It would be valuable to verify whether the conclusions reproduce under additional SID construction schemes (e.g., RQ-VAE) and additional backbones (TIGER, LC-Rec, LETTER).

  2. The contribution of reasoning needs to be demonstrated experimentally.

  3. The paper's premise is that the SID representations are obtained from pretraining on data, and in the experiments they are frozen rather than jointly trained. Does this setting arise in real-world application scenarios?

2 comments

We thank the reviewer for the comments. Before the individual points, we state the purpose of the paper, which the revision now makes explicit in the abstract and introduction. All changes are marked in blue.

  • Why reason: the rationale is the product of a reasoning reranker. Unlike reasoning in mathematics or coding, it is not a means to higher accuracy. At the final ranking stage it lets engineers inspect why a slate was ordered as it was, and it can be shown to users.
  • What the paper contributes: it defines reasoning reranking from SIDs alone as a task, evaluates existing models and common post-training methods on it, and exposes where they fall short. It does not propose a strong model. Its finding is that no current method is sufficient yet, while the pretrained backbone already holds the ranking signal, and it calls for post-training that closes this gap.

W1. The paper's central claim — that "SIDs carry behavioral signal that the caption text cannot provide" — is not actually established by the experiments. Because the SID representations are taken directly from the upstream pretrained version and kept frozen throughout, the authors never manipulate them as a variable in a controlled comparison; combined with the use of a single model (OpenOneRec) and a single dataset (Kuaishou), it is impossible to separate whether the gains come from a SID-specific signal or from priors already baked into that model's pretraining.

Q1. It would be valuable to verify whether the conclusions reproduce under additional SID construction schemes (e.g., RQ-VAE) and additional backbones (TIGER, LC-Rec, LETTER).

Response. We agree that the experiments do not isolate the SID representation, and the revision narrows the statement to the pretrained backbone. This leaves the contribution intact, because the paper does not propose a strong model or a new representation. It defines a task that is useful and has measurable headroom, evaluates current models and common post-training methods on it, and shows where they fall short. Three new controls make this concrete (new Table 3, new Appendix D.6).

  • Headroom: ranking the candidates by the pretrained 8B model's own likelihood of their SIDs reaches .415 NDCG@5, against .277 for the best post-trained reasoning model and .256 for the best text reader, whose prompts are about 20x longer.
  • A weakness of current methods: a count of how often each candidate occurs in other users' histories reaches .298, level with the best reasoning models. A strong text reader on the same slates is at chance with no history, so this prior is not in the captions.
  • What is open: history adds a personalized part that popularity cannot supply (+.109 in accuracy against random catalog items), but among items the platform exposed to the same user every system, text readers included, is at .55 or below, where chance is .50.

The claim is therefore that the pretrained backbone holds a ranking signal that item text does not provide, and that current methods carry only part of it into a reasoned ranking. Attributing it to the SID representation in isolation would require pretraining the backbone again with altered embeddings, which is beyond an evaluation of post-training.

On other SID schemes and backbones (Q1): the setting needs a backbone that reads SIDs and writes language, item text for the paired text view, and histories of up to 256 items. TIGER is not a language model, and the public LC-Rec and LETTER setups use review datasets with short histories. OpenOneRec is the one public release that meets these requirements. We release the testbed and code so that the study can be repeated on other backbones.

W2. The title foregrounds "Reasoning Reranking," yet the paper provides no evidence that the reasoning actually helps the ranking. During training the reward looks only at whether the final ranking is correct and not at the reasoning text at all (Section 5.1; Appendix C.4), and the authors run no "with-reasoning vs. without-reasoning" comparison. There is therefore no way to tell whether the improved ranking is attributable to the reasoning.

Q2. The contribution of reasoning needs to be demonstrated experimentally.

Response. We ran the comparison (new Appendix D.7). The 8B SFT and 8B RLVR models generate the ranked list after the rationale or directly, with the reasoning block closed empty, from the same prompts.

ModelHistoryWith rationaleWithoutDifference (95% CI)
8B SFTk=0.130.139+.009 (-.023, +.042)
8B SFTk=64.220.232+.011 (-.022, +.044)
8B RLVRk=0.187.175-.012 (-.040, +.015)
8B RLVRk=64.273.285+.011 (-.012, +.036)

No difference is significant, and the gain of RLVR over SFT is +.053 in both modes. Reasoning does not help the ranking, and the paper does not ask it to. The rationale is what a reasoning reranker delivers: an account of the ranking for engineers and users. The ranking reward trains the ranking, and the rationale is evaluated separately by its content (Section 5.2, new Appendix D.8).

That evaluation exposes the second weakness the paper reports: 63% of what the best model states about items is supported by the item text, against 94% for a caption reader. We report this as a finding about current methods and pose a faithful rationale as an open problem.

Q3. The paper's premise is that the SID representations are obtained from pretraining on data, and in the experiments they are frozen rather than jointly trained. Does this setting arise in real-world application scenarios?

Response. Yes. Adapting a released SID-pretrained checkpoint without changing its SID vocabulary is the common case. The item-to-code map and the token embeddings are shared by every component that reads or writes SIDs, so a downstream stage such as a reranker is trained on top of them and does not redefine them. Our methods follow this practice: they train LoRA adapters and leave the token embeddings unchanged (Section 5.1), which also gives every method the same starting representations.

The likelihood reference shows that the frozen embeddings are not the bottleneck: with the same embeddings the backbone ranks the slates at .415 NDCG@5. The gap is in how the reasoned ranking uses them.

Reviewer_fULVOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper asks whether a recommender pretrained on Semantic IDs (SIDs) can reason about and rerank a supplied slate without seeing item text. It introduces RERANKID, a paired testbed varying item rendering, history length, and slate size, and compares SFT, rejection-filtered training, on-policy distillation, and reward-based RL.

Critical strengths
  1. The same users and slates are evaluated with SID-only, text-only, and combined item views, with separate interventions on history and slate size. The paper also counts invalid outputs rather than silently excluding them (Section 3; Appendix B). This makes the reported ranking gain more credible than a simple cross-system comparison.

  2. The 8B graded-reward policy improves over its SFT initializer by +.040 NDCG@5 with a user-paired 95% interval of [+.006,+.076], while the binary-reward ablation does not improve over SFT (Table 2; Appendix D.1). SID prompts average 845 rather than 16,722 tokens at the anchor setting (Appendix D.2). Importantly, the paper also documents a major failure: item-claim accuracy favors the text reference 66:0 in the paired rationale audit (Table D.5). These results clarify both the benefit and the limitation of SID-only reranking.

Critical weaknesses
  1. At zero history, the model beats random, but this could reflect item popularity, the mixture of exposure-hard and random negatives, or content information encoded by the SIDs, rather than collaborative information unavailable from text. The comparison with a different text-reading model cannot separate these explanations (Section 5.2; Appendix A.2–A.4). A popularity or exposure-frequency baseline, and a control changing or removing behavior-trained SID embeddings, would materially narrow the mechanism claim.

  2. The paper's own audit finds that the SID policy mentions only about three of ten ranked candidates and often invents item details; the text reference wins item-claim accuracy 66:0 and overall rationale quality 97:0 (Section 5.2; Table D.5). The authors acknowledge this and suggest fetching metadata after ranking. Until that is tested, the contribution is best understood as compact SID-based ranking with verbalized but unreliable explanations, rather than a reliable reasoning reranker for users.

Questions for authors
  1. How much of the zero-history gain survives a matched item-popularity or exposure-frequency baseline, particularly when restricted to exposure-hard negatives? Is there evidence distinguishing behavioral SID embeddings from information inherited from the text-derived SID codes?

  2. If the rationale is shown to users, what fraction of item-specific statements are factually supported, and would retrieving metadata for just the selected items repair the error without losing the prompt-length advantage? If explanations are not meant to be trusted, please clarify the role of the generated rationale.

3 comments

We thank the reviewer for the constructive comments. Both requested baselines and both rationale tests are now in the paper. Before the individual points, we state the purpose of the paper, which the revision now makes explicit in the abstract and introduction. All changes are marked in blue.

  • Why reason: the rationale is the product of a reasoning reranker. Unlike reasoning in mathematics or coding, it is not a means to higher accuracy. At the final ranking stage it lets engineers inspect why a slate was ordered as it was, and it can be shown to users.
  • What the paper contributes: it defines reasoning reranking from SIDs alone as a task, evaluates existing models and common post-training methods on it, and exposes where they fall short. It does not propose a strong model. Its finding is that no current method is sufficient yet, while the pretrained backbone already holds the ranking signal, and it calls for post-training that closes this gap.

W1. At zero history, the model beats random, but this could reflect item popularity, the mixture of exposure-hard and random negatives, or content information encoded by the SIDs, rather than collaborative information unavailable from text. The comparison with a different text-reading model cannot separate these explanations (Section 5.2; Appendix A.2–A.4). A popularity or exposure-frequency baseline, and a control changing or removing behavior-trained SID embeddings, would materially narrow the mechanism claim.

Q1. How much of the zero-history gain survives a matched item-popularity or exposure-frequency baseline, particularly when restricted to exposure-hard negatives? Is there evidence distinguishing behavioral SID embeddings from information inherited from the text-derived SID codes?

Response. We added the popularity baseline and the split by negative type (new Table 3, new Appendix D.6). The last two columns give the probability that the positive is ranked above a random-fill negative and above an exposure-hard negative (chance is .500).

System, no historyNDCG@5vs. random fillvs. exposure-hard
Popularity.298.739.500
SID likelihood, 8B Base.330.808.527
8B RLVR.184.619.504
Gemini (text).084.503.492
  • Zero-history gain: it lies in the comparison with random fill and does not survive the restriction to exposure-hard negatives (.504, CI .474 to .534). The revision describes it as an item-level prior of the popularity or exposure kind, not as personalized preference (Section 5.2).
  • What history adds: the accuracy of 8B RLVR against random fill rises from .619 to .727 (+.109, CI +.075 to +.143), a personalized part that a popularity prior cannot supply. Against exposure-hard negatives no system exceeds .55, text readers included, and the revision reports this level as open.
  • Behavioral or content information: a caption reader on the same slates is at chance with no history, while a count of interactions reaches .298, so the prior is of behavioral origin and cannot be read from content. Separating the behavior-trained embeddings from the text-derived codes would require pretraining the backbone again, which is beyond a study of post-training, and the revision does not make that mechanism claim.

The narrower statement leaves the contribution intact. The paper defines the task, evaluates current models and methods on it, and reports their weaknesses, and none of this rests on the mechanism.

W2. The paper's own audit finds that the SID policy mentions only about three of ten ranked candidates and often invents item details; the text reference wins item-claim accuracy 66:0 and overall rationale quality 97:0 (Section 5.2; Table D.5). The authors acknowledge this and suggest fetching metadata after ranking. Until that is tested, the contribution is best understood as compact SID-based ranking with verbalized but unreliable explanations, rather than a reliable reasoning reranker for users.

Q2. If the rationale is shown to users, what fraction of item-specific statements are factually supported, and would retrieving metadata for just the selected items repair the error without losing the prompt-length advantage? If explanations are not meant to be trusted, please clarify the role of the generated rationale.

Response. We agree with this reading, and it is what the paper claims. We do not present a reliable reasoning reranker. We define the task, evaluate current methods, and report that rationales written from SIDs alone are not yet reliable (Section 5.2). We ran both tests the reviewer asks for (new Appendix D.8). The judge lists every statement a rationale makes about a named candidate and checks it against that candidate's caption.

RationaleItem statements supportedContradictedRationales with a contradicted statement
8B RLVR, from SIDs63.0%19.0%56%
Gemini, from captions94.3%0.8%6%
8B RLVR + metadata for its ten returned items98.9%0.2%1%
  • Retrieving metadata for the selected items: it repairs the error. A second call sees only the RLVR rationale, its ranking, and the captions of its ten returned candidates, and rewrites the rationale to agree with them. The supported share rises from 63% to 98.9%.
  • Prompt length: the second prompt is about 2.4k tokens, so the two prompts together are about 3.2k tokens against 16.7k for the text prompt, and the second prompt does not grow with history.
  • Role of the rationale: it is what the reranker delivers to engineers and users, and it is not a means to accuracy (removing it changes NDCG@5 by at most .012, new Appendix D.7). Making it faithful from SIDs alone is the open problem the paper poses, on top of a ranking signal that the backbone already holds (likelihood reference .415 against .256 for the best text reader).

Thank you for the additional experiments and clarifications. They address my main concerns, and I will maintain my positive recommendation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.