acceptodds
Under review as a conference paper at ICLR 2027

Understanding Retrieval and Reasoning in LLM Reranking under Availability Bias

Abstract

Large language models are increasingly used to rerank candidate items in recommender systems, where they may rely on both reasoning and retrieved evidence. We show that retrieval is not merely auxiliary to reasoning. Across two recommendation benchmarks, a small set of selected reviews matches a long reasoning trace on HR@1 at substantially lower inference cost, and the two signals combine. However, evaluating retrieval in this setting is confounded by a previously underappreciated bias. LLM rerankers systematically shift probability toward candidates that are accompanied by evidence-like text, even when that text belongs to another candidate or carries no useful information. We call this *availability bias*, by analogy to human judgment, without assuming a shared mechanism. Matching only the total number of retrieved passages does not remove the bias: when evidence is allocated unevenly, the allocation itself changes the ranking. We therefore introduce matched controls that separately account for evidence presence, count, allocation, content, and attribution. Under these controls, genuine evidence selection improves HR@1 by -, while much of the apparent advantage among common retrieval heuristics is explained by where they allocate evidence rather than by which passages they select. Our results show that retrieval is a first-class component of LLM reranking, but also a potential source of hidden ranking manipulation. Reliable evaluation must control for evidence availability and attribution, or retrieval systems may receive credit for effects unrelated to the information they retrieve.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Recommender Systems, Large Language Models, Reasoning, Information Retrieval

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_Tj9FOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies how retrieved reviews and reasoning affect LLM-based recommendation reranking. It identifies availability bias, where attaching evidence to candidates can influence rankings independently of the usefulness of that evidence, and proposes matched controls to separate the effects of evidence content, allocation, and attribution.

Critical strengths

The controlled experiments are carefully designed, particularly the content-resampled control that preserves each candidate’s evidence allocation while changing the selected passages.

Critical weaknesses
  1. Much of the allocation analysis in Sec. 6 and Table 4 relies on candidates competing for a shared passage budget across all items. However, this differs from approaches that retrieve evidence separately for each candidate. For example, RevBrowse [1] retrieves top-K features per item rather than across all candidates. The experiments establish that uneven allocation can influence rankings, but do not sufficiently demonstrate how often this confound occurs in existing systems or how substantially it affects their reported gains.

  2. In Sec. 2 and Appendix, requests are generated retrospectively from the target item’s own review, while negatives are mined around the known target. Such requests may emphasize item-specific attributes that are naturally recoverable from other reviews, making the task closer to matching product descriptions than predicting a user’s next choice. Excluding the source review and near-duplicates addresses direct leakage, but does not resolve this construction bias. Consequently, the retrieval-versus-reasoning conclusions in Sec. 3 require stronger validation using genuine pre-purchase requests and candidate lists produced by an actual first-stage recommender.

[1] Learning to Shop Like Humans: A Review-driven Retrieval-Augmented Recommendation Framework with LLMs. arxiv '25.

Questions for authors
  1. Do the main findings hold with unmodified human requests and candidates recalled from user history, without using the ground-truth target to construct the candidate set?

  2. How does reasoning affect availability bias itself? Does it reduce sensitivity to irrelevant or incorrectly attributed evidence, beyond improving overall ranking accuracy?

  3. Do the allocation findings remain consistent when evidence length is controlled per candidate, rather than matching only passage counts?

Reviewer_UhnAOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper examines the roles of retrieval and reasoning in LLM reranking, uncovering an availability bias: when candidate items are paired with auxiliary evidence like reviews, LLMs tend to inflate their ranking scores even if the evidence is mismatched or uninformative. To investigate this, the authors design matched controls to isolate the impacts of evidence presence, allocation, content, and attribution. Evaluating on Amazon-20 and Reddit recommendation datasets, the results show that selected short reviews achieve performance comparable to reasoning while using significantly fewer computational resources. Furthermore, controlling for evidence count yields a 0.04–0.10 HR@1 boost through evidence selection, though much of this gain actually stems from allocation rather than the retrieved content itself.

Critical strengths
  1. The authors use presence-matched, count-matched, content-resampled, and swapped controls to separate various confounding factors. This controlled-variable approach is far more convincing than simply checking the presence or absence of retrieval.

  2. The paper compares performance and inference costs between retrieval and reasoning, while analyzing the impacts of retrieval budgets, selection objectives, and evidence allocation strategies. Using paired bootstrap and multiple comparison corrections also reinforces the credibility of the findings.

Critical weaknesses
  1. In Amazon-20, requests are generated by paraphrasing target item reviews, and evidence is pulled from candidate review pools. Even after removing original and overlapping reviews, this setup creates strong semantic links between queries and target reviews, giving retrieval-based methods an edge. Additionally, the setup uses only two datasets (~750 samples each) with a fixed pool of 20 candidates per query, leaving open whether these findings hold up on real user queries.

  2. The swapped-evidence experiment tests for wrong attribution. However, Figure 2 tracks the overall probability mass of three candidates rather than individual rank shifts. When evidence is swapped, the total mass might stay similar even if the model fails to track who owns what. While other tests support misattribution, this set-level metric alone doesn't prove the model correctly links wrong evidence to specific candidates.

Questions for authors
  1. Can the authors report individual probability changes, ranking shifts, and HR@1 for each candidate after swapping evidence, instead of just the combined probability mass?

  2. Does availability bias show up in other model families, scales, or listwise/pairwise reranking methods?

  3. Do retrieval's efficiency and performance edges over reasoning hold steady across varying retrieval qualities, tighter reasoning budgets, and longer candidate lists?

Reviewer_fotZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies LLM reranking for recommendation. A Qwen3-8B reader ranks twenty candidates by the log-probability of each label, with or without a thinking trace. The tasks are two curated sets of 750 instances (Amazon-20 and Reddit-Movie) with popularity-matched negatives, and each candidate has a pool of reviews to retrieve from. The paper reports four findings. (1) Ten reviews chosen by a contrastive cosine rule, read without reasoning, are level with a ~2,800-token reasoning trace on HR@1 at about 0.25 s against a minute, and the two combine. (2) "Availability bias": text placed under the reader's top three candidates raises the odds that the answer is among them, even when each block describes another of the three; content-free filler does so part of the way. (3) Arbitrary passages at a matched count are worth nothing. (4) Against arbitrary passages at a matched count, selection rules gain +0.04 to +0.10 HR@1, and a content-resampled control attributes most of the shortlist gain of shared-budget rules to where the passages go rather than which passages are chosen.

Critical strengths

S1. Sound matched controls, and an interesting observation about uninformative text. The controls (presence-matched, count-matched, content-resampled, swapped) hold fixed what each selection rule was not allowed to decide. In particular, evidence-like text moves the reader toward the candidates it is attached to even when it carries little useful information: under the reader's top three, content-free filler of matched length moves the odds about a third as far as a real review (Fig. 2, Table 7). This shows that a with/without-retrieval comparison can credit retrieval for text it did not need, and the controls are a reusable way to separate the two.

S2. Careful and honest experiments. Every contrast is paired on the same instances, with bootstrap intervals and Holm correction (L141–147). The negatives are popularity-matched, so a popularity ranker drops from 0.341 to 0.068 HR@1 (L112), and negative results are reported openly, such as retrieval after reasoning buying nothing and arbitrary text at a matched count being worth nothing. Finding 1 holds in a weaker form: adding reasoning on top of retrieval helps in all four settings (+0.032 to +0.054, all intervals above zero; L1349). As a result, the measurements themselves are reliable.

Critical weaknesses

W1. The availability bias is an established phenomenon, so the contribution reduces to applying it to recommendation. (most concerning)

  • The paper presents as "a previously underappreciated bias" that "LLM rerankers systematically shift probability toward candidates that are accompanied by evidence-like text, even when that text belongs to another candidate or carries no useful information". Text attached to or injected into an item is already known to push LLM search engines, rerankers, and recommenders toward that item [1–3], mostly with crafted text. In a slightly different setting, LLMs are also known to be highly sensitive to position bias [4], and methods already address this kind of presentation sensitivity in LLM ranking [5]. Because this is an analysis paper rather than a methodology paper, the remaining contribution seem to be measuring a known effect in recommendation.

W2. The experimental results are hard to interpret.

  • The central evidence (Fig. 2) is a shift in the log-odds that the reader's answer falls in a three-candidate set, not a ranking metric, and the figure does not say what its two colors mean. HR is never defined, and most numbers are in 13 appendix tables.

W3. The results support the general claim only weakly.

  • Set-level measure: Fig. 2 measures the three-candidate set, and the swap only moves text among those three (L246–248), so a reader that credits the correct item would show the same real ≈ swapped result.
  • Uninformative text: the "no useful information" clause holds only under the reader's top three; under three random candidates filler lowers the odds in all four settings (Table 7).
  • Selection gain: the "+0.04–0.10 HR@1" credited to "genuine evidence selection" (L24–25) is mostly allocation by the paper's own split (+0.032 vs. +0.005 content; L1031–1034).
  • Scope: all availability results use 8B readers (Qwen3-8B; Llama-3.1-8B in App. D) on two curated sets of 750 instances. To make a broader claim about "LLM rerankers", need more models (with higher capabilities) is needed.
  • Cost comparison (Table 1, Fig. 1): the seconds-per-query column supports the claim that retrieval matches reasoning "at a fraction of the cost" (L59, L61, L481), but it seems a bit awkard, far from the paper's topic, which is availability bias and its controls. The result is also trivial (expected) by construction: a direct read generates no tokens, while a reasoning read generates 2,569 to 3,326 thinking tokens with greedy decoding (Table 1, "generated"; L99). The measurement is small: 20 queries per corpus, titles only, batch size one, on one idle H100, and the metadata and arbitrary-retrieval rows have no timing ("–", Table 1).

W4. Several main-text claims do not match the paper's own appendix or tables.

  • Sec. 6 says selection is worth +0.04 to +0.10 HR@1 on the reader's own shortlist (L393), but no rule in Table 6 exceeds +0.064; values near +0.10 appear only for the oracle or on the full list (Table 6, Table 14).
  • Sec. 3 says selected reviews are the stronger input at every size (L199–200), but at 8B on Reddit reasoning alone scores 0.375 and retrieval alone 0.368 (Table 13).
  • The intro says passages filed under the wrong candidates are worse than none (L67–68), but on Amazon-20 the re-filed arm is −0.020 [−0.051, +0.011] before the read and +0.008 with metadata (Table 11, L1395–1399).
  • App. D says the real-minus-swapped interval for reviews contains zero under every set and on both corpora (L1185–1186), but on Reddit under the bottom three it is +0.166 [+0.061, +0.278] (Table 7).
  • Tables 5 and 15 are captioned "presence-matched", but their shared-budget entries equal Table 6's count-matched rows.

References

[1] Aggarwal et al., GEO: Generative Engine Optimization, arXiv:2311.09735

[2] Tamber & Lin, Illusions of Relevance: Arbitrary Content Injection Attacks Deceive Retrievers, Rerankers, and LLM Judges, arXiv:2501.18536

[3] Zhang et al., Stealthy Attack on Large Language Model based Recommendation, arXiv:2402.14836

[4] Liu et al., Lost in the Middle: How Language Models Use Long Contexts, arXiv:2307.03172

[5] Tang et al., Found in the Middle: Permutation Self-Consistency Improves Listwise Ranking in Large Language Models, arXiv:2310.07712

Questions for authors
  1. Does the availability bias go away for larger, more capable readers? All availability experiments use 8B readers, and Table 13 varies reader size (1.7B–8B) only for the two levers, not for the bias. I suspect the effect may fade with more capable models. Is there an experiment that shows how the bias changes with model capability?
  2. Under the swap, does probability move onto the candidate whose heading the text sits under, or onto the candidate it describes? Every candidate's score is already read out (L98), so this needs only a reanalysis.
  3. Please reconcile the claims listed under W4 with the tables they cite. Please also clarify how the "arbitrary non-target" for the fabricated review was chosen (L266), given that App. D uses a candidate already top-1 in 0.286 of instances, and what "larger sizes" means for Llama (L263), given that every Llama shift under its own top three is smaller than Qwen's (Tables 7–8).
  4. Please define HR@k (hit rate) where it is first used. In Fig. 2, explain the difference between the blue and orange bars (why is "real, wrong candidate" orange?) and complete the caption, which ends with an incomplete sentence ("Qwen3-8B"). In Fig. 4 (App. D), fix the overlapping panel titles and value labels, which cannot be read.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.