- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how to select evidence under a token budget when a query has several relevant viewpoints. LMVU uses a small linear model to score passages by the viewpoints they add to the evidence already selected. Its features describe passage relevance, geometric relations to evidence and viewpoint atoms, and the selection state. The model uses frozen embeddings and pairwise supervision derived from viewpoint coverage. Experiments on four datasets measure coverage breadth and depth, with further analyses of budget, state conditioning, transfer, and oracle behavior. The evaluation focuses on evidence selection; it does not test downstream answer generation.
Critical strengths
-
Selecting passages by relevance alone can miss alternative answers or perspectives. LMVU addresses this with a linear model over frozen embeddings, which is inexpensive to use, and supervision based on marginal viewpoint coverage. The paper also acknowledges that its budget-feasible greedy procedure is a heuristic and does not claim an unproven approximation guarantee.
-
The evaluation provides useful detail beyond average performance. Table 1 shows better average breadth than MMR at the primary budget, while also showing where LMVU is not the best method. The appendix reports uncertainty and transfer results, and explains why state-constant features cancel in pairwise comparisons. The Perspective coverage-depth interval is above zero even though the breadth evidence is less conclusive.
Critical weaknesses
-
The objective gives equal weight to all annotated viewpoints, regardless of their prevalence or credibility. This is reasonable for queries asking for a range of perspectives, but may be less appropriate when users want the dominant view or the best-supported answer. The evaluation measures coverage of annotated viewpoints, without testing whether the resulting answers are more complete while still accurately reflecting the evidence. The paper should make the intended query scope clearer and avoid treating higher coverage as a general improvement in answer quality.
-
The experiments leave uncertainty about both practical efficiency and the size of the gains. LMVU updates features and rescores candidates after each selection, but the paper reports training time rather than selection latency or scaling with candidate-pool size. Some benchmarks also have few independent queries. In addition, JPR and RichRAG use adapted scoring procedures and consider only the top 64 candidates, while LMVU can access larger pools. These differences make it difficult to determine how much improvement comes from the learned ranking rule and what additional inference cost it requires.
Questions for authors
-
Which types of queries are intended to benefit from equal-weight viewpoint coverage? How should the method be used when viewpoint prevalence or evidence credibility matters? Please clarify whether the claim is limited to evidence coverage or also extends to final-answer quality. Evaluating downstream answers would be useful future work.
-
Are selection-time measurements available, including feature updates and repeated scoring? How does latency compare with static ranking and MMR as the candidate pool and number of selected passages grow? Please distinguish these measurements from the reported training times; a new large-scale benchmark is not required for the rebuttal.
-
Are results available with identical candidate access for LMVU, JPR, and RichRAG? Given the small evaluation sets, can the existing results also be summarized with paired uncertainty at the independent query or topic level?