- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper studies how retrieved reviews and reasoning affect LLM-based recommendation reranking. It identifies availability bias, where attaching evidence to candidates can influence rankings independently of the usefulness of that evidence, and proposes matched controls to separate the effects of evidence content, allocation, and attribution.
Critical strengths
The controlled experiments are carefully designed, particularly the content-resampled control that preserves each candidate’s evidence allocation while changing the selected passages.
Critical weaknesses
-
Much of the allocation analysis in Sec. 6 and Table 4 relies on candidates competing for a shared passage budget across all items. However, this differs from approaches that retrieve evidence separately for each candidate. For example, RevBrowse [1] retrieves top-K features per item rather than across all candidates. The experiments establish that uneven allocation can influence rankings, but do not sufficiently demonstrate how often this confound occurs in existing systems or how substantially it affects their reported gains.
-
In Sec. 2 and Appendix, requests are generated retrospectively from the target item’s own review, while negatives are mined around the known target. Such requests may emphasize item-specific attributes that are naturally recoverable from other reviews, making the task closer to matching product descriptions than predicting a user’s next choice. Excluding the source review and near-duplicates addresses direct leakage, but does not resolve this construction bias. Consequently, the retrieval-versus-reasoning conclusions in Sec. 3 require stronger validation using genuine pre-purchase requests and candidate lists produced by an actual first-stage recommender.
[1] Learning to Shop Like Humans: A Review-driven Retrieval-Augmented Recommendation Framework with LLMs. arxiv '25.
Questions for authors
-
Do the main findings hold with unmodified human requests and candidates recalled from user history, without using the ground-truth target to construct the candidate set?
-
How does reasoning affect availability bias itself? Does it reduce sensitivity to irrelevant or incorrectly attributed evidence, beyond improving overall ranking accuracy?
-
Do the allocation findings remain consistent when evidence length is controlled per candidate, rather than matching only passage counts?