- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies source preferences in LLM search agents. It compares retrieved results with matching requirement-satisfaction vectors while controlling display position. Source masking and identity swaps show that displayed source information affects selection. Controlled DPO experiments examine how correlations between source identity and item quality can create or reduce preferences, while additional experiments study missing information and prompting interventions.
Critical strengths
The paper combines an observational audit with convincing controlled interventions. The source-swap experiments are particularly useful because they show that changing the displayed source can change selection even when the item content is held fixed.
Critical weaknesses
The interpretation of "better" items remains limited by the satisfaction measure. The rubric captures requirements supported by snippets and omits some distinctions relevant to the original request, including certain accommodation qualities and scholarly qualifications. Consequently, matching on this rubric does not necessarily establish equal overall usefulness. The source-swap experiments convincingly establish source influence, but the inversion analysis is less conclusive about when that influence harms the user. A focused analysis of comparable source types and cases with more complete information would strengthen this interpretation.
The mitigation experiments establish changes in source selection, but their effect on decision quality remains unclear. Balanced DPO reduces source preference on equal-satisfaction pairs, but the paper does not report whether it preserves the ability to select the better item when requirement satisfaction differs. Appendix J describes unequal-satisfaction validation pairs, making their selection accuracy before and after DPO particularly informative. Without these results, the practical benefit of reducing the measured preference remains uncertain.
Questions for authors
Do the inversion results persist when comparing similar types of sources and when the available descriptions clearly establish whether the relevant requirements are satisfied, rather than leaving them unknown?
Could the authors report selection accuracy before and after balanced DPO on held-out pairs where one item satisfies an additional requirement and all other satisfaction labels agree, such as the validation pairs described in Appendix J?