- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces BasisDPP-RAG, a second-stage evidence-selection method intended to distinguish same-source repetition from corroboration across distinct evidence bases. The paper argues that conventional RAG selects passages primarily by individual relevance and may therefore allocate several limited context slots to passages that express similar claims derived from the same underlying source. This can exclude complementary evidence from another source that is necessary to answer the query.
For each retrieved candidate, BasisDPP-RAG constructs three views: the original passage, a query-conditioned claim span, and an evidence-basis representation. It learns relevance, claim, and basis representations and combines claim similarity and basis similarity in a quality-aware determinantal point process. The resulting set objective favors individually relevant passages while penalizing candidates that are similar in both claim and basis space. It can therefore suppress repeated formulations from one basis while retaining similar claims associated with different resolved bases.
The method is evaluated on AVeriTeC, NLI4CT, and SciFact using controlled candidate pools and a fixed evidence budget. The paper reports substantial improvements in Exact Match, independent-basis coverage, selected-basis redundancy, and robustness to distractors. The clinical-trial setting is particularly well matched to the method because the evidence bases are represented by stable trial identifiers and the queries commonly require evidence from one or two specified trials.
Overall, the paper makes a useful contribution to budgeted evidence selection when provenance is recoverable and the required evidence bases are present in the retrieved candidate pool. However, the current formulation assigns one principal claim view and one basis representation to each passage. Its applicability to multi-claim, multi-source documents, which is found more commonly in social media, where one document can draw from several underlying sources and several documents can echo one hidden upstream source, is not yet established by the reported experiments.
Critical strengths
The paper identifies and formalizes an important evidence-selection failure mode
The paper makes a consequential distinction between passage-level agreement and support from distinct evidential origins. A relevance-only selector can choose several accurate and highly relevant passages yet still produce an incomplete evidence set because all selected passages derive from one basis. This is particularly serious under a small passage or token budget.
The factorization of query-conditioned claim similarity and basis similarity is a meaningful improvement over generic semantic diversity. It allows the method to penalize same-claim, same-basis repetition without automatically discarding similar conclusions reached through different resolved bases or complementary claims contained within one basis.
Critical weaknesses
- The one-claim and one-basis representation limits applicability to multi-source, multi-claim documents
Each passage is represented using one principal query-conditioned claim span and one evidence basis. This is adequate for focused benchmark passages, clinical-trial sections, and scientific rationale sentences, but it may be too restrictive for open-domain documents.
For example, a news article may contain an official statement, a direct expert interview, a quotation taken from another publication, and statistics from a government report. These claims do not necessarily share one evidential basis. Conversely, several articles with different URLs may all repeat one upstream interview or press release. Treating the containing article as the basis can therefore over-merge distinct evidence within a document, while treating different article URLs as distinct bases can under-detect dependence across documents.
This weakness is consequential because the paper’s central claim concerns repetition versus corroboration, but the distinction depends on choosing the correct provenance granularity. The present experiments do not establish that this granularity can be recovered in naturally occurring multi-source documents.
- The experimental design supports a narrower conclusion than parts of the paper’s framing suggest
The experiments convincingly evaluate second-stage evidence selection from small, controlled candidate pools. However, candidate construction protects gold evidence and includes controlled same-basis descendants before producing the final six-candidate pool. This creates a setting that directly exposes the failure mode targeted by BasisDPP-RAG: several highly relevant formulations from one known basis compete with complementary evidence from another known basis.
This is a valid mechanism-level evaluation, but it does not fully test end-to-end retrieval or naturally occurring hidden dependence. If the first-stage retriever omits an important basis, BasisDPP-RAG cannot recover it. Similarly, controlled descendants with inherited basis information are easier to identify than syndicated articles, translations, paraphrases, or scientific papers sharing an unrecorded upstream dataset.
The results therefore strongly support the claim that basis-aware selection improves context allocation when basis identities are recoverable and the necessary bases are present. They do not yet establish a general solution to provenance-aware RAG across unrestricted multi-source documents. The paper should state this scope more explicitly and ideally include stronger diversity baselines or natural retrieval experiments.
Questions for authors
Candidate-pool construction: Could the authors state more explicitly how often controlled same-basis descendants appear in the principal test candidate pools, rather than only in training or diagnostic subsets? What is the performance of Standard RAG and BasisDPP-RAG when these controlled descendants are removed and candidate pools consist only of naturally occurring retrieved items?
Natural versus protected retrieval: How much of the reported improvement remains when candidate pools are constructed directly from the first-stage retriever without protecting gold evidence before truncation? Reporting passage recall, distinct-basis recall, and final accuracy as the candidate-pool size increases would help separate retrieval omissions from selection improvements.
Multi-claim passages: How frequently do passages in the three benchmarks contain more than one query-relevant claim? Has the paper measured how often the TopSpan representation omits a secondary claim that is required for the final decision?
Multi-basis passages: How would the current method represent a passage that contains claims derived from several origins, such as two expert interviews and an external report? Is one basis selected for the whole passage, or can the current implementation associate different spans with different bases?
7 comments
Q3. How frequent are multi-claim passages, and how often does TopSpan omit a required secondary claim?
In an independent audit of 200 naturally retrieved passages, 181 extracted spans retained all query-relevant information in the original passage. Six passages (3.0% of the audited sample) had extracted spans that omitted a necessary secondary proposition (Appendix Table 11). We did not separately count multi-claim passages whose relevant information was fully retained, so the audit measures observed extraction omissions without establishing multi-claim prevalence or the omission rate conditional on a passage containing multiple claims.
TopSpan retains the highest-scoring query-relevant sentence and may add one adjacent sentence to preserve a reference, negation, quantity, or qualification. This allows some multi-sentence content to be retained, but does not guarantee coverage of every relevant proposition in longer passages.
Q4. How would the method represent a passage with claims from several origins, and can different spans receive different bases?
The current implementation assigns one basis descriptor to each retrieved passage and does not associate different spans with different bases. Under the URL-based mapping, if only the containing article’s URL is available, a passage combining two expert interviews and an external report receives a single article-level basis descriptor, so the individual origins are not separately represented.
TopSpan likewise produces a single extracted span, which may include an adjacent sentence. Information outside that span is absent from the claim representation, although the original passage remains available to the relevance head and to the generator.
Conversely, articles with different URLs may repeat the same upstream source. If that common origin is not resolved, their URL-based keys remain distinct and their dependence may go undetected, so repeated evidence can be retained as apparent corroboration. Section 4.2 now states the absence of span-level source attribution explicitly. Source-homogeneous segmentation and multi-basis attribution are possible extensions for such documents.
5. Critical Weakness 1: provenance granularity in multi-source documents
Your examples identify two distinct challenges: separating multiple origins within one passage, and linking a shared origin across different passages. The current one-basis representation does not fully address either, and Appendix Table 21 discusses the corresponding risks of basis over-merging and provenance under-resolution.
Trial and paper identifiers provide explicit operational provenance units in NLI4CT and SciFact, while canonical URLs provide a coarser proxy in AVeriTeC. None of these identifiers alone guarantees independence from shared upstream data or reporting. The issue is especially visible in web articles, but it can also arise when one publication contains multiple studies or when several publications reuse the same underlying evidence.
We have clarified this in the scope of the paper: BasisDPP-RAG performs selection with respect to resolved evidence bases, rather than recovering the full provenance structure of unrestricted documents. The factorized kernel can preserve complementary claims within a resolved basis, but it cannot reconstruct multiple underlying origins that the representation does not encode.
6. Critical Weakness 2: scope and strength of the evaluation
We revised the framing and expanded the comparisons in line with your suggestions. The abstract now reports the 4.34-point gain over the strongest direct selector control under shared relevance scores in place of the previous framework-baseline comparison, and scopes the conclusion to settings where the required evidence bases are retrieved and provenance is recoverable—the condition you identify. The Table 1 caption now distinguishes the framework comparisons from the direct selector controls in Table 2, and the limitations state that the one-claim, one-basis representation cannot capture multi-source passages or shared upstream origins.
The added MMR, semantic DPP, provenance-deduplication, and hard-ID controls test the selection design against simpler alternatives. The no-descendants and natural-retrieval results further show that the gains are not confined to the original constructed pools.
We agree that inherited basis identifiers make the controlled descendants easier to group than naturally occurring hidden dependence. Natural retrieval removes the inserted descendants, but does not itself establish recovery of shared upstream sources. The revised experiments therefore extend the evidence for basis-aware selection beyond the original controlled setting, while retaining the distinction you identified between selecting evidence and resolving its provenance.
Thank you very much for the additional experiments and clearer scoping. The no-descendant and natural-retrieval results substantially address my concern that the gains were driven by constructed descendants or gold-evidence protection. The added matched controls are also helpful, especially hard-ID DPP: the remaining 2.3–2.8 point gains across AVeriTeC, NLI4CT, and SciFact suggest that BasisDPP-RAG provides value beyond exact provenance deduplication.
The response does not resolve the provenance-granularity issue, but it now states the limitation clearly. Each passage still receives one basis descriptor, while shared upstream origins across different URLs remain unresolved unless that provenance has already been identified. I therefore understand the revised contribution as basis-aware evidence selection when suitable evidence bases are available, rather than provenance discovery or a general defense against open-web narrative amplification.
The additional experiments, stronger matched controls, and narrower framing meaningfully addressed my earlier concens. While evidence on the statistical stability of the gains over hard-ID DPP would still be useful, I consider this a remaining limitation. I will maintain my current rating of weak accept.
Thank you for recognizing the distinction between repeated evidence and corroboration, and for identifying where the current representation needs a clearer scope. Your comments motivated additional selector controls, descendant-removal and natural-retrieval experiments, and an audit of claim-extraction completeness. We also revised the abstract and the limitations to separate the demonstrated selection benefit from the unresolved problem of provenance attribution in multi-source documents. We address your four questions below, followed by the two broader concerns.
Q1. How often do controlled descendants appear in the principal test pools, and how do Standard RAG and BasisDPP-RAG perform once they are removed?
Constructed same-basis descendants occur in 43.2% of queries and account for 12.8% of candidates in the principal controlled test pools—not only in training or diagnostic subsets—and the gains persist once they are removed.
Average Smart EM (%) with Llama-2-13B-Chat.
The original Standard RAG result is from Table 1; the direct selector comparisons are reported in Table 2. That baseline was not rerun in the two additional settings, and we distinguish it explicitly from shared-score Top-k, which uses the same trained relevance scores as the other direct controls. Against this matched relevance-only control, BasisDPP-RAG improves EM by 6.40 percentage points both without constructed descendants and under natural retrieval; its corresponding gains over hard-ID DPP are 2.70 and 2.47 points.
The no-descendants condition retains gold-evidence protection under the controlled protocol, whereas natural retrieval neither inserts constructed descendants nor protects gold evidence. Within each setting the direct controls share relevance scores, candidate pools, budgets, and generation settings; hard-ID retains our claim and quality terms while replacing learned basis similarity with smoothed ID equality.
Q2. How much of the improvement remains without gold protection, and can retrieval omissions be separated from selection?
Under natural retrieval without gold-evidence protection, BasisDPP-RAG remains 6.40 percentage points above shared-score Top-k and 2.47 points above hard-ID DPP at n=12, compared with margins of 8.37 and 4.34 points on controlled pools. We further compare against hard-ID DPP across three natural candidate-pool sizes, reporting retrieval coverage alongside answer accuracy:
Values are macro-averaged percentages across the three benchmarks, and EM includes all queries, including retrieval failures (new Table 3). As the pool grows from 6 to 24 candidates, retrieval coverage and EM both increase while BasisDPP-RAG maintains a 2.3–2.7 point advantage over hard-ID DPP. At n=12 the gains are positive on every dataset: 2.30 points on AVeriTeC, 2.80 on NLI4CT, and 2.30 on SciFact (Appendix Table 10).
These measurements report first-stage evidence availability separately from downstream accuracy; they do not by themselves attribute individual errors to retrieval or to selection. The controlled–natural comparison likewise does not isolate gold protection, because pool construction and pool size both differ. The results support better selection from retrieved evidence, and the selector cannot recover evidence absent from its candidate pool.