acceptodds
Under review as a conference paper at ICLR 2027

Submodular Evidence Selection for Retrieval-Augmented Generation

Abstract

Retrieval-augmented generation (RAG) conditions a frozen generator on a -passage subset of a larger candidate pool. Relevance-based selection suffers two failure modes that a larger budget relieves only inefficiently: facet starvation, where the top ranks over-serve one semantic dimension of the query while its remaining facets scatter at lower ranks, and higher-order redundancy, where pairwise-dissimilar passages jointly span a low-dimensional subspace and crowd out multi-hop evidence—in both cases, extra budget buys redundant copies rather than the missing evidence. We propose Submodular Evidence Selection (SES), which maximizes a single monotone submodular objective combining relevance-gated coverage—a frozen language model decomposes the query into sub-queries, and direct relevance is fused multiplicatively into per-facet coverage credit, so coverage is earned only by passages that themselves answer the query—with geometric diversity, the log-determinant volume spanned by the selected embeddings, which penalizes redundancy of any order. Both normalizers are constants fixed before optimization, so greedy maximization inherits the classical guarantee exactly, with no surrogate step. Across four multi-hop QA benchmarks and three frozen generators, SES consistently improves exact match over relevance-based and diversity-aware baselines, composes cleanly with cross-encoder reranking, and adds only millisecond-scale CPU selection beyond one cacheable decomposition call per query.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Evidence selection, Submodular optimization, Retrieval-augmented generation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_sbJmOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Submodular Evidence Selection (SES), which jointly optimize relevance-gated facet coverage and log-determinant geometric diversity. The objective is monotone submodular, so a greedy selection algorithm has a (1-1/e)-approximation guarantee. Empirically, SES shows promising improvements over baselines on multiple benchmarks.

Critical strengths
  1. The paper is well motivated and proposes a clean formulation. The formulation is intuitive and the monotone submodular objective comes with an approximation guarantee.

  2. The empirical results are promising and consistent.

Critical weaknesses
  1. Several closely related works are missing, including Chen et al. (NeurIPS 2018) on fast greedy DPP MAP inference, Lu and Sidiropoulos (COLM 2026) on principled and scalable diversity-aware retrieval, and Khan et al. (EACL Findings 2026) on query-aware relevance–diversity trade-offs. Chen et al. (NeurIPS 2018) is particularly relevant because its incremental DPP implementation suggests a more efficient way to maintain determinant-based gains (will detail later). Lu and Sidiropoulos (COLM 2026) report strong quality and scalability results, and a direct comparison would be useful. Khan et al. (EACL Findings 2026) is also closely related to the choice of the trade-off parameter.

  2. The current implementation has suboptimal time complexity. In Algorithm 1, Line 5 recomputes LS−1gjSL_S^{-1}g_{jS} for every remaining candidate, which costs O(∣S∣2)O(|S|^2) per candidate and leads to the O(nk3)O(nk^3) term. Eq. (8)–(9) in Chen et al. (NeurIPS 2018) cache and incrementally update LS−1gjSL_S^{-1}g_{jS} for each candidate jj, reducing the per-candidate update from O(∣S∣2)O(|S|^2) to O(∣S∣)O(|S|), suggesting that this part could be reduced to O(nk2)O(nk^2). In addition, the O(n2d)O(n^2d) full Gram precomputation may also be further optimized.

Reference:

Laming Chen, Guoxin Zhang, Hanning Zhou. Fast Greedy MAP Inference for Determinantal Point Process to Improve Recommendation Diversity. NeurIPS 2018.

Lu and Sidiropoulos. Principled and Scalable Diversity-Aware Retrieval via Cardinality-Constrained Binary Quadratic Programming. COLM 2026.

Saadat Hasan Khan, Spencer Hong, Jingyu Wu, Kevin Lybarger, Youbing Yin, Erin Babinsky, Daben Liu. DF-RAG: Query-Aware Diversity for Retrieval-Augmented Generation. EACL Findings 2026.

Questions for authors
  1. The paper only uses rerankers before SES. Have the authors considered applying a reranker after SES?

  2. Using log-det to capture higher-order redundancy is intuitive, but the empirical evaluation does not directly compare it with pairwise diversity measures such as ILAD or ILMD. Could the authors provide such a comparison to clarify whether log-det is more useful for RAG?

5 comments

Thank you for your constructive suggestions. Below, we address the related work, implementation improvements, and additional experiments.

W1. Related work and additional baseline comparisons (W1)

Chen et al. (NeurIPS 2018). Our DPP and SMART-RAG implementations use the greedy MAP inference procedure described by Chen et al. We will explicitly credit this work in the implementation description. Its cached projection updates also provide the basis for the SES optimization in Section 2.

CCBQP and DF-RAG. We have evaluated both suggested baselines on the same questions, 100-passage candidate pools, E5-base-v2 embeddings, and four-passage budget. For Lu and Sidiropoulos (COLM 2026), we use the authors’ Frank–Wolfe implementation of CCBQP with θ=0.9\theta=0.9. For Khan et al. (EACL Findings 2026), we use DF-RAG’s gMMR + Planner + Evaluator pipeline, selecting λDF∈{0.1,0.2,…,1.0}\lambda_{\mathrm{DF}}\in\{0.1,0.2,\ldots,1.0\} per query without reference answers. SES retains its shared λ=0.1\lambda=0.1.

All scores below are EM (%), with macro EM averaging the four datasets equally.

GeneratorMethod2WikiBamboogleHotpotQAStrategyQAMacro EM (%)
Qwen2.5-14B-InstructCCBQP / FW (θ=0.9\theta=0.9)29.419.235.064.237.0
Qwen2.5-14B-InstructDF-RAG28.121.634.864.837.3
Qwen2.5-14B-InstructSES28.822.434.565.537.8
Qwen2.5-7B-InstructCCBQP / FW (θ=0.9\theta=0.9)26.518.431.965.135.5
Qwen2.5-7B-InstructDF-RAG27.811.231.664.733.8
Qwen2.5-7B-InstructSES27.021.632.766.236.9
Llama-3.1-8B-InstructCCBQP / FW (θ=0.9\theta=0.9)24.911.230.963.832.7
Llama-3.1-8B-InstructDF-RAG23.512.030.362.232.0
Llama-3.1-8B-InstructSES25.516.032.163.434.3

SES leads both added baselines in macro EM under every generator. This extends the comparison to a quadratic-programming approach and a query-adaptive relevance–diversity method, providing further support for SES’s joint coverage-and-diversity design.

W2. Cached marginal updates and on-demand Gram columns (W2)

We have implemented the suggested optimization, reducing geometric-update work from O(nk3)O(nk^3) to O(nk2)O(nk^2) without changing the selected passages. Here, nn, kk, mm, and dd denote candidate-pool size, passage budget, number of query facets, and embedding dimension.

Following Chen et al.’s incremental updates, we cache each candidate’s projection coefficients and Schur residual. With tt passages already selected, updating a candidate then costs O(t+1)O(t+1) instead of solving a triangular system in O(t2)O(t^2). Summed over the greedy steps, this gives O(nk2)O(nk^2) geometric work and O(nk)O(nk) cache storage. Coverage scoring contributes O(nkm)O(nkm).

We also compute the Gram column G:i=EeiG_{:i}=Ee_i when passage ii is selected. At most kk columns are needed for selection:

Selection componentOriginal implementationUpdated implementation
Geometric marginal updatesO(nk3)O(nk^3)O(nk2)O(nk^2)
Kernel construction for selectionO(n2d)O(n^2d)O(nkd)O(nkd)
Gram/projection storage for selection, excluding embeddingsO(n2)O(n^2)O(nk)O(nk)

The resulting selection-phase cost is O(nkd+nkm+nk2)O(nkd+nkm+nk^2), given facet scores and the original fixed normalizers. Facet-score construction and spectral normalization, including the Gram computation used by the current normalizer, are accounted for separately as preprocessing. We will update Algorithm 1 and the complexity analysis to describe the cache and separate selection costs from preprocessing.

Q3. Applying a reranker after SES

We considered post-SES reranking, but chose to incorporate reranker scores before SES so that relevance and evidence coverage inform the same selection decision. SES selects passages jointly: a passage with a lower individual relevance score may still supply an otherwise missing fact. Pruning the selected set by individual reranker scores can remove such complementary evidence and undo the coverage achieved by SES.

Following the reviewer’s suggestion, we tested this design choice by letting SES select eight passages and then applying MiniLM or BGE to retain four. We use Qwen2.5-14B-Instruct, the original 3,125 questions, and λ=0.1\lambda=0.1. The table also includes reranker-only selection, which ranks the original 100-passage pool and takes the top four, and our default reranker → SES pipeline, which uses reranker scores as the relevance gate while SES selects four passages from that pool. Every pipeline supplies four passages to the generator.

Pipeline2WikiBamboogleHotpotQAStrategyQAMacro EM (%)
SES → 428.822.434.565.537.8
MiniLM → 4 (reranker only)29.219.236.664.637.4
MiniLM → SES → 429.620.037.465.238.1
SES → 8 → MiniLM → 430.416.837.065.037.3
BGE → 4 (reranker only)32.220.840.962.139.0
BGE → SES → 432.222.441.462.339.6
SES → 8 → BGE → 429.722.437.364.038.4

Post-SES pruning falls below direct reranker-only selection in macro EM for both rerankers: 37.3% versus 37.4% for MiniLM, and 38.4% versus 39.0% for BGE. By contrast, incorporating reranker scores before SES yields the highest macro EM within each reranker group: 38.1% with MiniLM and 39.6% with BGE. Although post-SES BGE improves over SES alone, it does not improve over direct BGE reranking in these results.

These findings support our choice of integrating reranker relevance into SES’s joint objective. The reranker-only and pre-SES rows reproduce the original paper results; the post-SES rows and SES-only control are from the supplementary comparison. The tested post-SES configuration selects eight passages and prunes to four.

Q2. Comparing log-det with ILAD and ILMD

Log-det achieves the highest macro EM among the three diversity variants and matches or exceeds both pairwise alternatives on every dataset in this comparison. We replace only the diversity term, retaining the relevance gate, query coverage, candidate pools, decompositions, E5 embeddings, four-passage budget, Qwen2.5-14B-Instruct generator, and generation settings. All diversity variants use λ=0.1\lambda=0.1.

We define passage distance as d(i,j)=(1−cos⁡(ei,ej))/2d(i,j)=(1-\cos(e_i,e_j))/2. ILAD sums pairwise distances and divides by the fixed final-budget count (k2)\binom{k}{2}. ILMD uses the minimum pairwise distance, with value zero for sets containing fewer than two passages. Selection uses each objective’s actual greedy marginal, including negative ILMD marginals.

Configuration2WikiBamboogleHotpotQAStrategyQAMacro EM (%)
Gated coverage-only27.721.635.664.837.4
Coverage + ILAD28.822.433.565.237.5
Coverage + ILMD28.217.631.864.235.5
SES / log-det28.822.434.565.537.8

Log-det reaches 37.8% macro EM, compared with 37.5% for ILAD and 35.5% for ILMD. Against ILAD, it ties on 2Wiki and Bamboogle and improves HotpotQA and StrategyQA by 1.0 and 0.3 points, respectively. This supports log-det as an effective complement to query coverage under the shared setting. The choice also preserves the method’s theoretical structure. Log-det evaluates a candidate’s regularized residual relative to the selected embeddings, accounting for collective geometric dependence. For the paper’s log-det form, nonnegative facet scores, fixed positive normalizers, ϵ>0\epsilon>0, and λ≥0\lambda\geq0, the combined objective is normalized, monotone, and submodular. Greedy selection therefore retains its (1−1/e)(1-1/e) objective-value guarantee. The fixed-denominator ILAD extension has increasing pairwise marginal gains, while ILMD can decrease when a passage is added; their coverage-augmented objectives do not generally inherit the same guarantee.

References:

Laming Chen, Guoxin Zhang, Hanning Zhou. Fast Greedy MAP Inference for Determinantal Point Process to Improve Recommendation Diversity. NeurIPS 2018.

Qiheng Lu, Nicholas D. Sidiropoulos. Principled and Scalable Diversity-Aware Retrieval via Cardinality-Constrained Binary Quadratic Programming. COLM 2026.

Saadat Hasan Khan, Spencer Hong, Jingyu Wu, Kevin Lybarger, Youbing Yin, Erin Babinsky, Daben Liu. DF-RAG: Query-Aware Diversity for Retrieval-Augmented Generation. EACL Findings 2026.

Thank you for the detailed response and the additional experiments. I appreciate the authors' efforts in addressing my concerns.

Since ICLR allows manuscript revisions during the discussion period, I suggest incorporating the optimized algorithm and updated complexity analysis, as well as the additional experimental results, into the revised manuscript.

Reviewer_UMaPOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

I particularly like the core idea of this paper. Combining relevance-gated facet coverage with geometric diversity is an interesting and well-motivated modeling choice, and the submodular formulation provides a clear optimization framework. However, the empirical analysis is substantially less developed than the formulation. This is disappointing because the reported overall improvements suggest potential, but the experiments do not sufficiently explain why the method works. The empirical results should not show the overall performance but why the method actually works (the theoretical analysis is good, but I just want to understand whether the theoretical assumptions really match the real applications). Not just ablation, but deeper insights to explicitly show why the method works.

Critical strengths
  • I particularly like the core idea. Combining relevance-gated facet coverage with geometric diversity is an interesting and coherent way to model evidence selection, addressing both query coverage and redundancy within the selected set.
  • The submodular formulation provides a clear optimization framework with an approximation guarantee for the proposed objective. The reported improvements across benchmarks and compatibility with cross-encoder reranking also suggest practical value.
Critical weaknesses
  • My major concerns are that the author does not give a sufficient empirical analysis of why the method works. Specifically, The gap between geometric diversity and semantic redundancy: Linear dependence among passage embeddings does not necessarily imply redundant evidence, and geometric novelty does not necessarily imply useful new information. In Section 4.4, the 0.4-point average EM improvement from adding diversity is interpreted as evidence that it captures redundancy missed by coverage. This result supports the usefulness of the component, but does not directly validate the proposed mechanism.

  • Several analyses would strengthen the paper:

  1. Measure whether geometric marginal gains correspond to genuinely new facts or supporting evidence, rather than differences in wording or topic.
  2. Provide cases where pairwise similarity fails to identify collectively redundant passages, while the log-determinant term succeeds.
  3. Explain when diversity helps or hurts. HotpotQA performs best without the diversity term. Relating these differences to candidate-pool redundancy and evidence requirements would provide useful insight into the method’s strengths and limitations.
  4. Similar passages may contain distinct facts needed for a multi-hop answer. Analyze whether geometric diversity preserves such evidence or sometimes suppresses useful complementary passages.
Questions for authors
  • Do geometric marginal gains correlate with the amount of genuinely new supporting information a passage provides?

  • Can you demonstrate cases where pairwise similarity misses collective redundancy but the log-determinant term successfully identifies it?

  • Why does HotpotQA perform best without the diversity term? Can this be explained by differences in evidence requirements or candidate-pool redundancy?

7 comments

Thank you for the suggestions on understanding why SES works. Below, we present the additional analyses and case studies addressing your questions.

Relating geometric diversity to semantic redundancy requires considering both the representation and the information needed for the task. Determinantal selection models diversity through a kernel over item representations and balances it with item quality (Kulesza and Taskar, 2011). In multi-document summarization, Cho et al. (2019) improve the similarity measure within a DPP specifically to capture redundant information expressed in different words. In RAG, Khan et al. (2026) find that the best relevance–diversity balance varies across datasets and develop query-aware selection. These findings motivate examining geometric diversity in relation to the evidence required by each query.

SES combines relevance-gated query coverage and geometric diversity. Coverage rewards passages addressing the query’s different requirements, with diminishing returns for repeatedly covering the same requirement. Geometry adds a preference for diversity among the selected documents. Whether that preference supplies useful evidence depends on what a passage adds to the current selection and which facts the answer requires.

We examine the contribution of geometry by comparing SES with gated coverage-only under otherwise identical settings. We measure the supporting facts retained in the selected set and the resulting answer changes, then inspect cases where the selection fills a missing requirement or preserves complementary facts. Regressions help identify when the same diversity preference can disrupt the evidence needed for an answer.

All analyses use Qwen2.5-14B-Instruct, the same 100-passage candidate pools, a four-passage budget, and fixed decompositions, embeddings, and generation settings. Gated coverage-only retains the relevance gate and sets λ=0.0\lambda=0.0; full SES uses λ=0.1\lambda=0.1 on every dataset. Top-k uses the cached relevance ranking, and MMR adds pairwise similarity-based redundancy control.

W1/Q1. Does adding geometric diversity improve supporting-evidence coverage?

Adding geometric diversity increases support recall on both 2Wiki and HotpotQA. To examine the reviewer's question about useful new information, we compare the annotated support retained by SES and gated coverage-only. This comparison measures whether adding the geometric term improves the evidence available to the generator.

We identify annotated supporting sentences by source title and sentence index. After normalizing Unicode, case, punctuation, and whitespace, we require the complete sentence to occur contiguously in the passage body, with a matching title. This measures the presence of annotated support; unmatched paraphrases remain unverified.

For each question, support recall is the fraction of distinct support units recoverable from the candidate pool that appear in the selected four passages. All methods are evaluated on the same 689 alignable 2Wiki questions and 708 alignable HotpotQA questions.

Selector2Wiki support recall (%)HotpotQA support recall (%)
Top-k74.666.4
MMR75.666.9
Gated coverage-only78.673.9
SES80.274.7

SES improves support recall over MMR by 4.6 points on 2Wiki and 7.8 on HotpotQA. Gated coverage-only already contributes gains of 3.0 and 7.0 points; adding geometry yields a further 1.6 and 0.8 points. This controlled comparison provides evidence beyond answer EM for the two-term design: query coverage recovers more of the required support, and geometry further improves the evidence retained under the same four-passage budget.

These results establish an observed benefit from adding geometry to evidence selection; a direct correlation between the geometric marginal and the number of new supporting facts remains unestablished. We will distinguish these claims in Section 4.4.

Support diagnostics are limited to datasets with usable annotations under this matching protocol. For StrategyQA, it recovers only 13 facts across 12 of 1,000 questions; Bamboogle lacks comparable supporting-fact annotations. Answer-level comparisons below retain all four datasets.

References

Alex Kulesza, Ben Taskar. Learning Determinantal Point Processes. UAI 2011.

Sangwoo Cho, Logan Lebanoff, Hassan Foroosh, Fei Liu. Improving the Similarity Measure of Determinantal Point Processes for Extractive Multi-Document Summarization. ACL 2019.

Saadat Hasan Khan, Spencer Hong, Jingyu Wu, Kevin Lybarger, Youbing Yin, Erin Babinsky, Daben Liu. DF-RAG: Query-Aware Diversity for Retrieval-Augmented Generation. EACL Findings 2026.

W2/Q2. Can log-det identify collective redundancy missed by pairwise similarity?

We provide two cases showing how SES recovers evidence missed by top-k relevance ranking and MMR’s pairwise redundancy penalty. All methods select four passages from the same candidate pool.

Case 1 — SES versus top-k: “Which band was formed first, Wire or Young the Giant?” (HotpotQA). The required facts are that Wire was formed in October 1976 and Young the Giant in 2004. Top-k selects passages about the album On a Wire, the different band Fighting with Wire, and two passages about Young the Giant. It therefore covers only one of the two annotated supporting sentences, missing the formation date of the intended band Wire. SES selects Wire (band) from rank seven alongside the Young the Giant evidence, covers both supporting sentences, and correctly answers Wire. The decomposed queries request each band’s formation date, allowing coverage to reward the missing entity–date requirement.

Case 2 — SES versus MMR: “Which film was released more recently, Drugstore Cowboy or When the Sea Rises?” (2Wiki). The required release years are 1989 and 2004, respectively. MMR selects two passages about Drugstore Cowboy and two about the band Drugstore, which took its name from that film. Despite the pairwise redundancy penalty, all four passages concern material associated with one side of the comparison. The evidence for When the Sea Rises is missing, and the model incorrectly answers Drugstore Cowboy. SES includes the passage identifying When the Sea Rises as a 2004 film, covers both dates, and correctly answers When the Sea Rises.

W3/Q3. When does diversity help or hurt, and why does HotpotQA differ?

Adding geometry improves EM on three of four datasets and raises the macro average under a single global weight. The following comparison changes only λ\lambda, retaining the relevance gate and all other settings. Wrong → correct and correct → wrong count answer changes from gated coverage-only to SES.

DatasetQuestionsGated coverage-only EM (%)SES EM (%)Gain (pp)Wrong → correctCorrect → wrong
2Wiki1,00027.728.8+1.13928
Bamboogle12521.622.4+0.821
HotpotQA1,00035.634.5−1.12738
StrategyQA1,00064.865.5+0.72518

The macro average rises from 37.4% to 37.8%. Gains on 2Wiki, Bamboogle, and StrategyQA are 1.1, 0.8, and 0.7 points, respectively. HotpotQA is the exception, reducing the overall average gain. These results support geometric diversity as a useful addition to query coverage while showing that its benefit depends on the evidence requirements.

Successful changes fill a missing requirement. For a 2Wiki question asking for the mother of the performer of Passion, SES replaces a repeated song passage with Utada Hikaru’s biography, which supplies Keiko Fuji and changes the answer from incorrect to correct. HotpotQA also has successful changes: a question linking Double Jeopardy to Ashley Judd is answered correctly after SES adds a biography giving the duration of her acting career as “more than two decades.”

Limitation: a fixed geometric weight can displace an essential fact or introduce a competing entity. For the date of Clemson’s 37–17 victory over Oklahoma at Sun Life Stadium, SES replaces the passage giving December 31, 2015 with a passage about the Miami Hurricanes playing at the stadium, and the model abstains. In another question, SES retains the link from Nevada Wolf Pack to the University of Nevada, Reno, but adds material about NC State Wolfpack football; the model then answers North Carolina State University.

The HotpotQA breakdown locates the net loss in bridge questions:

Question typeQuestionsWrong → correctCorrect → wrongNet change in correct answers
Bridge7952234−12
Comparison20554+1

Bridge questions require connecting facts through an intermediate entity. Similar passages may supply different necessary links, so replacing one with more novel material can break the chain. The Bach example below illustrates this failure directly. The counts locate the decline, while the cases identify possible mechanisms; they do not establish that candidate-pool redundancy differs systematically across datasets.

HotpotQA’s support recall rises from 73.9% to 74.7% even as EM falls. The two measurements concern different outcomes and populations: recall averages recoverable support over 708 alignable questions, whereas EM covers all 1,000 questions. More supporting units on average also need not mean a complete reasoning chain or an unambiguous context for each answer. The evidence-loss and entity-confusion cases illustrate why these metrics can move in different directions.

W4. Does SES preserve similar passages containing complementary facts?

SES preserves similar but complementary passages in more questions than top-k or MMR in this diagnostic. We identify passage pairs whose cosine similarity is at or above the within-query 90th percentile and where each passage contains an aligned supporting unit absent from the other. We then count eligible questions for which the selected set retains at least one such pair.

Selector2Wiki: questions retaining a pair / 92 eligibleHotpotQA: questions retaining a pair / 87 eligible
Top-k1734
MMR1733
SES1841

On HotpotQA, SES retains such pairs in 41 of 87 eligible questions, compared with 34 for top-k and 33 for MMR; on 2Wiki, the corresponding counts are 18 versus 17 and 17. Thus, the joint objective can preserve complementary evidence even when the passages are highly similar. These counts compare complete selectors; they do not attribute the preservation advantage to geometry alone.

For the question about the population of the county served by Collier Area Transit, one selected Collier County, Florida passage reports the 2000 population. SES also retains a similar passage giving the 2010 census population of 321,520, the reference answer. Its coverage advantage outweighs another candidate’s geometric advantage, and the model answers correctly.

Limitation: preservation is a trade-off, not a hard constraint. For “Who is the paternal grandfather of Johann Nicolaus Bach?”, two similar biographies supply separate links: Johann Nicolaus → Johann Christoph → Heinrich Bach. SES omits the second-hop biography, selects other Bach-family material, and answers Johannes Bach. This example identifies a specific boundary of the current objective: its geometric preference can outweigh the coverage of a necessary link.

The added evidence supports SES at both the evidence and answer levels: it recovers more annotated support than top-k and MMR, benefits from adding geometry on both support-recall benchmarks and three EM benchmarks, and can retain similar passages carrying different required facts. The main practical limitation observed here is that a fixed diversity weight can disrupt tightly linked evidence in some bridge questions. We will incorporate these results and this limitation into the revision, and distinguish the demonstrated benefits of the joint objective from claims about geometry as a standalone measure of semantic redundancy.

Reviewer_THg6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes SES, a training-free RAG selector combining relevance-gated coverage and log-determinant diversity with a greedy approximation guarantee. Across four QA benchmarks and three generators, SES improves average exact match, including with reranking.

Critical strengths
  1. The objective combines coverage and diversity with explicit marginal gains and a greedy approximation guarantee, making selection easy to implement.
  2. SES improves average EM across all three generators, including after reranking, supporting the value of passage composition.
Critical weaknesses
  1. The relevance gate lacks an ablation, and diversity adds only 0.4 average EM points while hurting HotpotQA, leaving its contribution unclear.
  2. Gains over the strongest baseline are only 0.9–1.5 average EM points, with no uncertainty estimates. The Qwen-14B gain over GeoRAG on Bamboogle is just one correct answer, leaving reliability uncertain.
Questions for authors
  1. With other settings fixed, does removing the relevance gate reduce performance?
  2. Are SES’s gains over the strongest baseline and coverage-only selection statistically significant?
3 comments

Thank you for the questions about the ablation and the reliability of the gains. We have added the requested gate ablation and paired-bootstrap analysis using Qwen2.5-14B-Instruct.

W1/Q1. Ablation: contributions of relevance gating and geometric diversity

SES combines relevance-gated query coverage and geometric diversity. The coverage term rewards passages that address the query’s different requirements, with the gate anchoring those matches to the original question. The geometric term adds a preference for diversity among the selected documents. To examine their contributions, we remove the gate or the geometric term separately, keeping the other settings fixed.

Configuration2WikiBamboogleHotpotQAStrategyQAMacro EM (%)
Full SES28.822.434.565.537.8
Without relevance gate28.020.031.264.736.0
Without geometric diversity27.721.635.664.837.4

Removing the relevance gate lowers performance on all four datasets. We set the original-query relevance weights to 1, retain λ=0.1\lambda=0.1, and recompute the coverage normalizer using the same definition. The candidate pool, cached subqueries, embeddings, passage budget, and generation settings remain fixed. Macro EM falls from 37.8% to 36.0%, with the largest loss on HotpotQA (3.3 points). These results support weighting facet matches by relevance to the full question.

Geometric diversity improves three datasets, while HotpotQA offsets part of that benefit. Removing diversity retains the gate and sets λ=0.0\lambda=0.0. Adding it back improves 2Wiki by 1.1 points, Bamboogle by 0.8, and StrategyQA by 0.7, but lowers HotpotQA by 1.1. The 0.4-point macro gain therefore averages positive contributions on three datasets and a negative contribution on one. Full SES uses the same λ=0.1\lambda=0.1 on all datasets, without dataset-specific adjustment.

We examined the HotpotQA answer changes to understand this difference. Of its 1,000 questions, 795 are bridge questions and 205 are comparison questions:

HotpotQA question typeWrong → correct after adding diversityCorrect → wrong after adding diversity
Bridge2234
Comparison54

The decline is concentrated in bridge questions, which require connecting facts through an intermediate entity. Passages about related entities can be geometrically similar while supplying different, necessary links. For example, answering “Who is the paternal grandfather of Johann Nicolaus Bach?” requires both Johann Nicolaus → Johann Christoph and Johann Christoph → Heinrich Bach. Gated coverage-only retains both biographies and answers correctly. SES replaces the second-hop biography with other Bach-family material and answers Johannes Bach. In this case, favoring geometric novelty breaks the evidence chain.

Thus, the two terms play different roles: query coverage rewards evidence for the required facts, while geometric diversity changes the composition of the selected documents. The latter improves EM on three benchmarks, but can displace complementary evidence in some HotpotQA bridge questions. Full SES still improves HotpotQA over top-k (34.5% versus 32.3%) and GeoRAG (34.1%).

W2/Q2. Statistical significance of the gains over the strongest baseline

The pooled improvement over GeoRAG is statistically significant under the paired-bootstrap analysis. GeoRAG is the strongest baseline by macro EM for Qwen2.5-14B-Instruct. SES improves macro EM from 36.9% to 37.8%, with gains of 1.4, 0.8, 0.4, and 1.1 points on 2Wiki, Bamboogle, HotpotQA, and StrategyQA, respectively.

The EM scores (%) for the methods included in this analysis are shown below.

Method / configuration2Wiki (n=1,000)Bamboogle (n=125)HotpotQA (n=1,000)StrategyQA (n=1,000)Macro EM (%)
SES (gate on, λ=0.1\lambda=0.1)28.822.434.565.537.8
GeoRAG27.421.634.164.436.9
Ungated coverage-only27.821.633.864.837.0

We resample the 3,125 evaluation questions with replacement for 8,000 repetitions, using seed 42 and identical sampled indices for SES and the comparator. We compute the paired EM difference in each repetition and report its percentile interval.

ComparatorMacro EM gain (pp)Pooled EM gain (pp)95% CI for pooled gain (pp)
GeoRAG+0.9+1.0[0.1, 2.2]
Ungated coverage-only+0.8+0.8[0.1, 1.9]

Both intervals exclude zero. These intervals concern pooled per-question EM, which weights datasets by sample count; macro EM gives each dataset equal weight. The significance claim applies to the pooled comparison under this evaluation setup. The paired analysis evaluates the improvement across all 3,125 questions, including positive gains on the other three datasets, rather than relying on that single answer.

We hope this message finds you well. We wanted to kindly follow up on our previous response and ask whether our revisions and explanations have fully addressed all of your concerns. If you feel that the remaining issues have been satisfactorily resolved, we would be grateful if you would consider updating your assessment accordingly. Thank you very much for your time and valuable feedback.

We would like to kindly follow up on our responses and revisions. We would appreciate hearing whether they have addressed your concerns, and we would be happy to provide any further clarification.

Thank you again for your time and valuable feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.