- Rating
- 3
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Submodular Evidence Selection (SES), which jointly optimize relevance-gated facet coverage and log-determinant geometric diversity. The objective is monotone submodular, so a greedy selection algorithm has a (1-1/e)-approximation guarantee. Empirically, SES shows promising improvements over baselines on multiple benchmarks.
Critical strengths
-
The paper is well motivated and proposes a clean formulation. The formulation is intuitive and the monotone submodular objective comes with an approximation guarantee.
-
The empirical results are promising and consistent.
Critical weaknesses
-
Several closely related works are missing, including Chen et al. (NeurIPS 2018) on fast greedy DPP MAP inference, Lu and Sidiropoulos (COLM 2026) on principled and scalable diversity-aware retrieval, and Khan et al. (EACL Findings 2026) on query-aware relevance–diversity trade-offs. Chen et al. (NeurIPS 2018) is particularly relevant because its incremental DPP implementation suggests a more efficient way to maintain determinant-based gains (will detail later). Lu and Sidiropoulos (COLM 2026) report strong quality and scalability results, and a direct comparison would be useful. Khan et al. (EACL Findings 2026) is also closely related to the choice of the trade-off parameter.
-
The current implementation has suboptimal time complexity. In Algorithm 1, Line 5 recomputes for every remaining candidate, which costs per candidate and leads to the term. Eq. (8)–(9) in Chen et al. (NeurIPS 2018) cache and incrementally update for each candidate , reducing the per-candidate update from to , suggesting that this part could be reduced to . In addition, the full Gram precomputation may also be further optimized.
Reference:
Laming Chen, Guoxin Zhang, Hanning Zhou. Fast Greedy MAP Inference for Determinantal Point Process to Improve Recommendation Diversity. NeurIPS 2018.
Lu and Sidiropoulos. Principled and Scalable Diversity-Aware Retrieval via Cardinality-Constrained Binary Quadratic Programming. COLM 2026.
Saadat Hasan Khan, Spencer Hong, Jingyu Wu, Kevin Lybarger, Youbing Yin, Erin Babinsky, Daben Liu. DF-RAG: Query-Aware Diversity for Retrieval-Augmented Generation. EACL Findings 2026.
Questions for authors
-
The paper only uses rerankers before SES. Have the authors considered applying a reranker after SES?
-
Using log-det to capture higher-order redundancy is intuitive, but the empirical evaluation does not directly compare it with pairwise diversity measures such as ILAD or ILMD. Could the authors provide such a comparison to clarify whether log-det is more useful for RAG?
5 comments
Q3. Applying a reranker after SES
We considered post-SES reranking, but chose to incorporate reranker scores before SES so that relevance and evidence coverage inform the same selection decision. SES selects passages jointly: a passage with a lower individual relevance score may still supply an otherwise missing fact. Pruning the selected set by individual reranker scores can remove such complementary evidence and undo the coverage achieved by SES.
Following the reviewer’s suggestion, we tested this design choice by letting SES select eight passages and then applying MiniLM or BGE to retain four. We use Qwen2.5-14B-Instruct, the original 3,125 questions, and . The table also includes reranker-only selection, which ranks the original 100-passage pool and takes the top four, and our default reranker → SES pipeline, which uses reranker scores as the relevance gate while SES selects four passages from that pool. Every pipeline supplies four passages to the generator.
| Pipeline | 2Wiki | Bamboogle | HotpotQA | StrategyQA | Macro EM (%) |
|---|---|---|---|---|---|
| SES → 4 | 28.8 | 22.4 | 34.5 | 65.5 | 37.8 |
| MiniLM → 4 (reranker only) | 29.2 | 19.2 | 36.6 | 64.6 | 37.4 |
| MiniLM → SES → 4 | 29.6 | 20.0 | 37.4 | 65.2 | 38.1 |
| SES → 8 → MiniLM → 4 | 30.4 | 16.8 | 37.0 | 65.0 | 37.3 |
| BGE → 4 (reranker only) | 32.2 | 20.8 | 40.9 | 62.1 | 39.0 |
| BGE → SES → 4 | 32.2 | 22.4 | 41.4 | 62.3 | 39.6 |
| SES → 8 → BGE → 4 | 29.7 | 22.4 | 37.3 | 64.0 | 38.4 |
Post-SES pruning falls below direct reranker-only selection in macro EM for both rerankers: 37.3% versus 37.4% for MiniLM, and 38.4% versus 39.0% for BGE. By contrast, incorporating reranker scores before SES yields the highest macro EM within each reranker group: 38.1% with MiniLM and 39.6% with BGE. Although post-SES BGE improves over SES alone, it does not improve over direct BGE reranking in these results.
These findings support our choice of integrating reranker relevance into SES’s joint objective. The reranker-only and pre-SES rows reproduce the original paper results; the post-SES rows and SES-only control are from the supplementary comparison. The tested post-SES configuration selects eight passages and prunes to four.
Q2. Comparing log-det with ILAD and ILMD
Log-det achieves the highest macro EM among the three diversity variants and matches or exceeds both pairwise alternatives on every dataset in this comparison. We replace only the diversity term, retaining the relevance gate, query coverage, candidate pools, decompositions, E5 embeddings, four-passage budget, Qwen2.5-14B-Instruct generator, and generation settings. All diversity variants use .
We define passage distance as . ILAD sums pairwise distances and divides by the fixed final-budget count . ILMD uses the minimum pairwise distance, with value zero for sets containing fewer than two passages. Selection uses each objective’s actual greedy marginal, including negative ILMD marginals.
| Configuration | 2Wiki | Bamboogle | HotpotQA | StrategyQA | Macro EM (%) |
|---|---|---|---|---|---|
| Gated coverage-only | 27.7 | 21.6 | 35.6 | 64.8 | 37.4 |
| Coverage + ILAD | 28.8 | 22.4 | 33.5 | 65.2 | 37.5 |
| Coverage + ILMD | 28.2 | 17.6 | 31.8 | 64.2 | 35.5 |
| SES / log-det | 28.8 | 22.4 | 34.5 | 65.5 | 37.8 |
Log-det reaches 37.8% macro EM, compared with 37.5% for ILAD and 35.5% for ILMD. Against ILAD, it ties on 2Wiki and Bamboogle and improves HotpotQA and StrategyQA by 1.0 and 0.3 points, respectively. This supports log-det as an effective complement to query coverage under the shared setting. The choice also preserves the method’s theoretical structure. Log-det evaluates a candidate’s regularized residual relative to the selected embeddings, accounting for collective geometric dependence. For the paper’s log-det form, nonnegative facet scores, fixed positive normalizers, , and , the combined objective is normalized, monotone, and submodular. Greedy selection therefore retains its objective-value guarantee. The fixed-denominator ILAD extension has increasing pairwise marginal gains, while ILMD can decrease when a passage is added; their coverage-augmented objectives do not generally inherit the same guarantee.
References:
Laming Chen, Guoxin Zhang, Hanning Zhou. Fast Greedy MAP Inference for Determinantal Point Process to Improve Recommendation Diversity. NeurIPS 2018.
Qiheng Lu, Nicholas D. Sidiropoulos. Principled and Scalable Diversity-Aware Retrieval via Cardinality-Constrained Binary Quadratic Programming. COLM 2026.
Saadat Hasan Khan, Spencer Hong, Jingyu Wu, Kevin Lybarger, Youbing Yin, Erin Babinsky, Daben Liu. DF-RAG: Query-Aware Diversity for Retrieval-Augmented Generation. EACL Findings 2026.
Thank you for the detailed response and the additional experiments. I appreciate the authors' efforts in addressing my concerns.
Since ICLR allows manuscript revisions during the discussion period, I suggest incorporating the optimized algorithm and updated complexity analysis, as well as the additional experimental results, into the revised manuscript.
Thank you for your constructive suggestions. Below, we address the related work, implementation improvements, and additional experiments.
W1. Related work and additional baseline comparisons (W1)
Chen et al. (NeurIPS 2018). Our DPP and SMART-RAG implementations use the greedy MAP inference procedure described by Chen et al. We will explicitly credit this work in the implementation description. Its cached projection updates also provide the basis for the SES optimization in Section 2.
CCBQP and DF-RAG. We have evaluated both suggested baselines on the same questions, 100-passage candidate pools, E5-base-v2 embeddings, and four-passage budget. For Lu and Sidiropoulos (COLM 2026), we use the authors’ Frank–Wolfe implementation of CCBQP with θ=0.9. For Khan et al. (EACL Findings 2026), we use DF-RAG’s gMMR + Planner + Evaluator pipeline, selecting λDF∈{0.1,0.2,…,1.0} per query without reference answers. SES retains its shared λ=0.1.
All scores below are EM (%), with macro EM averaging the four datasets equally.
SES leads both added baselines in macro EM under every generator. This extends the comparison to a quadratic-programming approach and a query-adaptive relevance–diversity method, providing further support for SES’s joint coverage-and-diversity design.
W2. Cached marginal updates and on-demand Gram columns (W2)
We have implemented the suggested optimization, reducing geometric-update work from O(nk3) to O(nk2) without changing the selected passages. Here, n, k, m, and d denote candidate-pool size, passage budget, number of query facets, and embedding dimension.
Following Chen et al.’s incremental updates, we cache each candidate’s projection coefficients and Schur residual. With t passages already selected, updating a candidate then costs O(t+1) instead of solving a triangular system in O(t2). Summed over the greedy steps, this gives O(nk2) geometric work and O(nk) cache storage. Coverage scoring contributes O(nkm).
We also compute the Gram column G:i=Eei when passage i is selected. At most k columns are needed for selection:
The resulting selection-phase cost is O(nkd+nkm+nk2), given facet scores and the original fixed normalizers. Facet-score construction and spectral normalization, including the Gram computation used by the current normalizer, are accounted for separately as preprocessing. We will update Algorithm 1 and the complexity analysis to describe the cache and separate selection costs from preprocessing.