acceptodds
Under review as a conference paper at ICLR 2027

Repetition or Corroboration? Evidence Selection for Retrieval-Augmented Generation

Abstract

Retrieval-augmented generation (RAG) ranks passages by relevance, but agreement among passages does not guarantee independent evidential support. Several passages may repeat one evidence basis, causing redundant support and discarding useful corroboration from distinct sources. We introduce BasisDPP-RAG, which separates the query-conditioned claim from its evidence basis and combines them in a factorized, quality-aware determinantal point process (DPP) for budgeted evidence selection. The resulting objective penalizes same-basis repetition while preserving agreement across distinct bases. Experiments on AVeriTeC, NLI4CT, and SciFact yield 84.84% average Exact Match, 11.17 percentage points above the strongest baseline, while reducing NLI4CT basis redundancy from 57.76% to 7.89% and improving robustness to adversarial distractors. These results show that reliable RAG should measure evidential support by distinct bases rather than passage count.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Retrieval-Augmented Generation, Determinantal Point Process, Evidence Selection, Source Dependence, Fact Verification

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_tBonOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces BasisDPP-RAG, a second-stage evidence-selection method intended to distinguish same-source repetition from corroboration across distinct evidence bases. The paper argues that conventional RAG selects passages primarily by individual relevance and may therefore allocate several limited context slots to passages that express similar claims derived from the same underlying source. This can exclude complementary evidence from another source that is necessary to answer the query.

For each retrieved candidate, BasisDPP-RAG constructs three views: the original passage, a query-conditioned claim span, and an evidence-basis representation. It learns relevance, claim, and basis representations and combines claim similarity and basis similarity in a quality-aware determinantal point process. The resulting set objective favors individually relevant passages while penalizing candidates that are similar in both claim and basis space. It can therefore suppress repeated formulations from one basis while retaining similar claims associated with different resolved bases.

The method is evaluated on AVeriTeC, NLI4CT, and SciFact using controlled candidate pools and a fixed evidence budget. The paper reports substantial improvements in Exact Match, independent-basis coverage, selected-basis redundancy, and robustness to distractors. The clinical-trial setting is particularly well matched to the method because the evidence bases are represented by stable trial identifiers and the queries commonly require evidence from one or two specified trials.

Overall, the paper makes a useful contribution to budgeted evidence selection when provenance is recoverable and the required evidence bases are present in the retrieved candidate pool. However, the current formulation assigns one principal claim view and one basis representation to each passage. Its applicability to multi-claim, multi-source documents, which is found more commonly in social media, where one document can draw from several underlying sources and several documents can echo one hidden upstream source, is not yet established by the reported experiments.

Critical strengths

The paper identifies and formalizes an important evidence-selection failure mode

The paper makes a consequential distinction between passage-level agreement and support from distinct evidential origins. A relevance-only selector can choose several accurate and highly relevant passages yet still produce an incomplete evidence set because all selected passages derive from one basis. This is particularly serious under a small passage or token budget.

The factorization of query-conditioned claim similarity and basis similarity is a meaningful improvement over generic semantic diversity. It allows the method to penalize same-claim, same-basis repetition without automatically discarding similar conclusions reached through different resolved bases or complementary claims contained within one basis.

Critical weaknesses
  1. The one-claim and one-basis representation limits applicability to multi-source, multi-claim documents

Each passage is represented using one principal query-conditioned claim span and one evidence basis. This is adequate for focused benchmark passages, clinical-trial sections, and scientific rationale sentences, but it may be too restrictive for open-domain documents.

For example, a news article may contain an official statement, a direct expert interview, a quotation taken from another publication, and statistics from a government report. These claims do not necessarily share one evidential basis. Conversely, several articles with different URLs may all repeat one upstream interview or press release. Treating the containing article as the basis can therefore over-merge distinct evidence within a document, while treating different article URLs as distinct bases can under-detect dependence across documents.

This weakness is consequential because the paper’s central claim concerns repetition versus corroboration, but the distinction depends on choosing the correct provenance granularity. The present experiments do not establish that this granularity can be recovered in naturally occurring multi-source documents.

  1. The experimental design supports a narrower conclusion than parts of the paper’s framing suggest

The experiments convincingly evaluate second-stage evidence selection from small, controlled candidate pools. However, candidate construction protects gold evidence and includes controlled same-basis descendants before producing the final six-candidate pool. This creates a setting that directly exposes the failure mode targeted by BasisDPP-RAG: several highly relevant formulations from one known basis compete with complementary evidence from another known basis.

This is a valid mechanism-level evaluation, but it does not fully test end-to-end retrieval or naturally occurring hidden dependence. If the first-stage retriever omits an important basis, BasisDPP-RAG cannot recover it. Similarly, controlled descendants with inherited basis information are easier to identify than syndicated articles, translations, paraphrases, or scientific papers sharing an unrecorded upstream dataset.

The results therefore strongly support the claim that basis-aware selection improves context allocation when basis identities are recoverable and the necessary bases are present. They do not yet establish a general solution to provenance-aware RAG across unrestricted multi-source documents. The paper should state this scope more explicitly and ideally include stronger diversity baselines or natural retrieval experiments.

Questions for authors

Candidate-pool construction: Could the authors state more explicitly how often controlled same-basis descendants appear in the principal test candidate pools, rather than only in training or diagnostic subsets? What is the performance of Standard RAG and BasisDPP-RAG when these controlled descendants are removed and candidate pools consist only of naturally occurring retrieved items?

Natural versus protected retrieval: How much of the reported improvement remains when candidate pools are constructed directly from the first-stage retriever without protecting gold evidence before truncation? Reporting passage recall, distinct-basis recall, and final accuracy as the candidate-pool size increases would help separate retrieval omissions from selection improvements.

Multi-claim passages: How frequently do passages in the three benchmarks contain more than one query-relevant claim? Has the paper measured how often the TopSpan representation omits a secondary claim that is required for the final decision?

Multi-basis passages: How would the current method represent a passage that contains claims derived from several origins, such as two expert interviews and an external report? Is one basis selected for the whole passage, or can the current implementation associate different spans with different bases?

7 comments

Thank you for recognizing the distinction between repeated evidence and corroboration, and for identifying where the current representation needs a clearer scope. Your comments motivated additional selector controls, descendant-removal and natural-retrieval experiments, and an audit of claim-extraction completeness. We also revised the abstract and the limitations to separate the demonstrated selection benefit from the unresolved problem of provenance attribution in multi-source documents. We address your four questions below, followed by the two broader concerns.

Q1. How often do controlled descendants appear in the principal test pools, and how do Standard RAG and BasisDPP-RAG perform once they are removed?

Constructed same-basis descendants occur in 43.2% of queries and account for 12.8% of candidates in the principal controlled test pools—not only in training or diagnostic subsets—and the gains persist once they are removed.

Average Smart EM (%) with Llama-2-13B-Chat.

MethodControlledNo descendantsNatural (n=12n=12)
Original Standard RAG45.97——
Shared-score Top-kk76.4773.1063.80
MMR77.4774.0065.40
Semantic DPP77.8774.4065.90
Rule deduplication78.8775.5066.60
Hard-ID DPP80.5076.8067.73
BasisDPP-RAG84.8479.5070.20

The original Standard RAG result is from Table 1; the direct selector comparisons are reported in Table 2. That baseline was not rerun in the two additional settings, and we distinguish it explicitly from shared-score Top-kk, which uses the same trained relevance scores as the other direct controls. Against this matched relevance-only control, BasisDPP-RAG improves EM by 6.40 percentage points both without constructed descendants and under natural retrieval; its corresponding gains over hard-ID DPP are 2.70 and 2.47 points.

The no-descendants condition retains gold-evidence protection under the controlled protocol, whereas natural retrieval neither inserts constructed descendants nor protects gold evidence. Within each setting the direct controls share relevance scores, candidate pools, budgets, and generation settings; hard-ID retains our claim and quality terms while replacing learned basis similarity with smoothed ID equality.

Q2. How much of the improvement remains without gold protection, and can retrieval omissions be separated from selection?

Under natural retrieval without gold-evidence protection, BasisDPP-RAG remains 6.40 percentage points above shared-score Top-kk and 2.47 points above hard-ID DPP at n=12n=12, compared with margins of 8.37 and 4.34 points on controlled pools. We further compare against hard-ID DPP across three natural candidate-pool sizes, reporting retrieval coverage alongside answer accuracy:

Candidates nnPassage recallBasis recallAll bases foundHard-ID EMBasisDPP-RAG EM
671.560.251.462.564.8
1282.173.564.767.770.2
2488.081.473.670.473.1

Values are macro-averaged percentages across the three benchmarks, and EM includes all queries, including retrieval failures (new Table 3). As the pool grows from 6 to 24 candidates, retrieval coverage and EM both increase while BasisDPP-RAG maintains a 2.3–2.7 point advantage over hard-ID DPP. At n=12n=12 the gains are positive on every dataset: 2.30 points on AVeriTeC, 2.80 on NLI4CT, and 2.30 on SciFact (Appendix Table 10).

These measurements report first-stage evidence availability separately from downstream accuracy; they do not by themselves attribute individual errors to retrieval or to selection. The controlled–natural comparison likewise does not isolate gold protection, because pool construction and pool size both differ. The results support better selection from retrieved evidence, and the selector cannot recover evidence absent from its candidate pool.

Q3. How frequent are multi-claim passages, and how often does TopSpan omit a required secondary claim?

In an independent audit of 200 naturally retrieved passages, 181 extracted spans retained all query-relevant information in the original passage. Six passages (3.0% of the audited sample) had extracted spans that omitted a necessary secondary proposition (Appendix Table 11). We did not separately count multi-claim passages whose relevant information was fully retained, so the audit measures observed extraction omissions without establishing multi-claim prevalence or the omission rate conditional on a passage containing multiple claims.

TopSpan retains the highest-scoring query-relevant sentence and may add one adjacent sentence to preserve a reference, negation, quantity, or qualification. This allows some multi-sentence content to be retained, but does not guarantee coverage of every relevant proposition in longer passages.

Q4. How would the method represent a passage with claims from several origins, and can different spans receive different bases?

The current implementation assigns one basis descriptor to each retrieved passage and does not associate different spans with different bases. Under the URL-based mapping, if only the containing article’s URL is available, a passage combining two expert interviews and an external report receives a single article-level basis descriptor, so the individual origins are not separately represented.

TopSpan likewise produces a single extracted span, which may include an adjacent sentence. Information outside that span is absent from the claim representation, although the original passage remains available to the relevance head and to the generator.

Conversely, articles with different URLs may repeat the same upstream source. If that common origin is not resolved, their URL-based keys remain distinct and their dependence may go undetected, so repeated evidence can be retained as apparent corroboration. Section 4.2 now states the absence of span-level source attribution explicitly. Source-homogeneous segmentation and multi-basis attribution are possible extensions for such documents.

5. Critical Weakness 1: provenance granularity in multi-source documents

Your examples identify two distinct challenges: separating multiple origins within one passage, and linking a shared origin across different passages. The current one-basis representation does not fully address either, and Appendix Table 21 discusses the corresponding risks of basis over-merging and provenance under-resolution.

Trial and paper identifiers provide explicit operational provenance units in NLI4CT and SciFact, while canonical URLs provide a coarser proxy in AVeriTeC. None of these identifiers alone guarantees independence from shared upstream data or reporting. The issue is especially visible in web articles, but it can also arise when one publication contains multiple studies or when several publications reuse the same underlying evidence.

We have clarified this in the scope of the paper: BasisDPP-RAG performs selection with respect to resolved evidence bases, rather than recovering the full provenance structure of unrestricted documents. The factorized kernel can preserve complementary claims within a resolved basis, but it cannot reconstruct multiple underlying origins that the representation does not encode.

6. Critical Weakness 2: scope and strength of the evaluation

We revised the framing and expanded the comparisons in line with your suggestions. The abstract now reports the 4.34-point gain over the strongest direct selector control under shared relevance scores in place of the previous framework-baseline comparison, and scopes the conclusion to settings where the required evidence bases are retrieved and provenance is recoverable—the condition you identify. The Table 1 caption now distinguishes the framework comparisons from the direct selector controls in Table 2, and the limitations state that the one-claim, one-basis representation cannot capture multi-source passages or shared upstream origins.

The added MMR, semantic DPP, provenance-deduplication, and hard-ID controls test the selection design against simpler alternatives. The no-descendants and natural-retrieval results further show that the gains are not confined to the original constructed pools.

We agree that inherited basis identifiers make the controlled descendants easier to group than naturally occurring hidden dependence. Natural retrieval removes the inserted descendants, but does not itself establish recovery of shared upstream sources. The revised experiments therefore extend the evidence for basis-aware selection beyond the original controlled setting, while retaining the distinction you identified between selecting evidence and resolving its provenance.

Thank you very much for the additional experiments and clearer scoping. The no-descendant and natural-retrieval results substantially address my concern that the gains were driven by constructed descendants or gold-evidence protection. The added matched controls are also helpful, especially hard-ID DPP: the remaining 2.3–2.8 point gains across AVeriTeC, NLI4CT, and SciFact suggest that BasisDPP-RAG provides value beyond exact provenance deduplication.

The response does not resolve the provenance-granularity issue, but it now states the limitation clearly. Each passage still receives one basis descriptor, while shared upstream origins across different URLs remain unresolved unless that provenance has already been identified. I therefore understand the revised contribution as basis-aware evidence selection when suitable evidence bases are available, rather than provenance discovery or a general defense against open-web narrative amplification.

The additional experiments, stronger matched controls, and narrower framing meaningfully addressed my earlier concens. While evidence on the statistical stability of the gains over hard-ID DPP would still be useful, I consider this a remaining limitation. I will maintain my current rating of weak accept.

Reviewer_yeKSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes BasisDPP-RAG to distinguish redundant evidence fromindependent corroboration in RAG. It learns query-conditioned claim andevidence-basis representations, combines their similarities in a factorized DPPkernel, and selects evidence under a context budget. Experiments on AVeriTeC,NLI4CT, and SciFact report 84.84% average EM, with improvements in evidencediversity and distractor robustness.

Critical strengths
  1. Meaningful problem formulation. The paper highlights an important distinction between repeated information and independent evidential support, which is not adequately captured by conventional relevance-based selection.
  2. Coherent formulation and promising results. The Claim–Basis factorization provides an interpretable way to model evidence redundancy. The reported improvements and ablations support its effectiveness under the evaluated settings.
Critical weaknesses
  1. Insufficient validation of Claim–Basis modeling. The method relies on a fixed sentence extractor and provenance identifiers to construct claim and basis representations. However, claim extraction quality and basis-resolution accuracy are not independently evaluated. In particular, distinct source IDs do not necessarily imply independent evidence, while basis-pair supervision is derived from the same identifiers provided to the encoder. The construction of claim-pair labels and valid evidence subsets also requires clearer justification. Consequently, it remains unclear whether the learned representations capture generalizable evidence dependence or primarily encode benchmark-specific source identities.
  2. Limited direct comparisons and potentially favorable evaluation settings The paper lacks direct comparisons with MMR, conventional DPP, and rule-based provenance deduplication. Existing ablations do not establish whether the learned factorization outperforms these simpler alternatives. Moreover, candidate pools incorporate gold evidence, controlled same-basis descendants, and hard negatives before selecting three passages from six candidates. Although this ensures controlled comparisons, it may favor the targeted redundancy mechanism. Results on naturally retrieved candidates are needed to support broader generalization claims.
Questions for authors

Q1. How are claim-pair labels and valid evidence subsets constructed? Can the authors provide independent evaluations of claim extraction and basis resolution? Q2. How does the method compare against MMR, standard DPP, and rule-based source deduplication under identical candidate pools and budgets? Q3. Does the improvement persist on naturally retrieved candidate pools without controlled same-basis descendants, particularly when provenance metadata is incomplete? Q4. How are unresolved sources handled in the inference-time basis kernel, given that they share the [UNK BASIS] representation?

5 comments

Thank you for these concerns; they guided the revision. To address Critical Weakness 1, we clarified the supervision targets and added independent audits of claim extraction and source-dependence judgments. For Critical Weakness 2, we added the requested selector baselines and a hard-ID control, together with evaluations without constructed descendants and under natural retrieval, plus a provenance-descriptor masking test. Below we answer Q1–Q4, then discuss what the evidence establishes for each critical weakness.

Q1. How are labels and valid sets constructed, and are the representations independently evaluated? (Critical Weakness 1)

In response, we clarified the supervision targets (Appendix Table 9) and added independent audits of extracted claim spans and source-dependence judgments (Appendix Table 11).

A claim-pair positive requires the same query-relevant proposition, not merely the same verdict; two passages addressing different required propositions are complementary and are not claim-pair positives. Basis-pair positives share a resolved primary origin, different resolved origins are negatives, and unresolved pairs are masked from Lbasis\mathcal{L}_{\mathrm{basis}} rather than labeled independent. A valid size-kk set covers the benchmark’s annotated evidence requirements — source-linked answers for AVeriTeC, trial evidence for NLI4CT, cited-abstract rationales for SciFact. Constructed descendants inherit their parent’s basis and cannot supply an additional required origin. Queries with no valid size-kk set contribute no set loss, while their item and pair losses remain defined. Appendix B.6 illustrates these definitions and the valid-set criterion with a held-out NLI4CT example. The addition therefore fixes the supervision targets and their exclusion conditions; it does not yet provide a complete operational specification of label and valid-set construction.

For the audits we examined 200 naturally retrieved passages and 200 source pairs. Reference judgments were made independently of the training labels and without access to resolver outputs, using primary-source evidence rather than ID equality.

Independent auditComplete spansCorrect / resolvable pairsUnresolved pairs
AVeriTeC62/7053/6010
NLI4CT65/7061/655
SciFact54/6048/555
Total181/200162/18020

A span is complete if it retains all query-relevant information in the original passage, rather than answering the entire query on its own. Six of the 19 incomplete spans omit a necessary second proposition. The relation errors comprise seven false merges and 11 false splits; unresolved pairs are excluded from the accuracy denominator. As clarified in Appendix B.6, these aggregate relation counts do not separately establish resolver and learned-kernel accuracy.

Q2. How does the method compare with simpler selectors under matched conditions? (Critical Weakness 2)

All three alternatives you name are now reported as direct selector controls under one shared relevance signal, together with a hard-ID factorized DPP, and the learned factorization exceeds every one of them in all three candidate-pool settings.

Average Smart EM (%)ControlledNo descendantsNatural (n=12n{=}12)
Supervised Top-kk76.4773.1063.80
MMR77.4774.0065.40
Semantic DPP77.8774.4065.90
Rule deduplication78.8775.5066.60
Hard-ID DPP80.5076.8067.73
BasisDPP-RAG84.8479.5070.20

BasisDPP-RAG exceeds the strongest direct control by 4.34, 2.70, and 2.47 percentage points respectively (new Table 2), with per-dataset results in Appendix Table 10. On controlled pools, generic diversity improves EM by 1.0–1.4 points over relevance-only selection, explicit provenance grouping by 2.4–4.0, and the learned factorization by 8.4; the same ordering of the six selectors holds in all three settings. This addresses the gap you identified in the ablations.

On the matching: the controls share the candidate pools, our trained relevance scores rir_i, a limit of three selected passages, the token budget, the frozen generator, the prompt, and the decoding rule, while differing in their selection rules and in their similarity or provenance inputs. MMR combines rir_i with passage cosine similarity. Semantic DPP is the conventional single-view DPP: one normalized passage-similarity kernel with the same quality transform and the same budgeted greedy selector as the full method. Rule deduplication groups only resolved matching basis keys, keeps the highest-relevance item per group, and fills any remaining slots by relevance. Hard-ID DPP retains our learned claim kernel and quality weights but replaces the learned basis kernel with Kijb=η+(1−η)1[gi=gj]K^b_{ij}=\eta+(1-\eta)\mathbf{1}[g_i=g_j] for resolved basis keys, with unresolved items grouped into a shared category; η\eta is tuned on development data over [0,1][0,1], whose endpoint η=0\eta=0 is exact ID equality. Parameters requiring tuning are selected on development data (Appendix Table 8). These are matched selection-rule comparisons, not equal-supervision retraining of every framework: the controls reuse our relevance head, and hard-ID additionally reuses our claim head.

Q3. Does the improvement persist on natural pools and with incomplete provenance? (Critical Weakness 2)

On natural pools, BasisDPP-RAG outperforms hard-ID DPP at all three evaluated pool sizes. For incomplete provenance, the added stress test masks descriptors after source resolution and therefore does not establish robustness to genuinely missing identifiers.

Natural pools contain neither inserted descendants nor protected gold evidence, and we score all queries, including retrieval failures:

Pool size nnPassage recallBasis recallAll bases foundHard-ID EMBasisDPP-RAG EM
671.560.251.462.564.8
1282.173.564.767.770.2
2488.081.473.670.473.1

Values are macro-averaged percentages across the three datasets (new Table 3). Incomplete retrieval limits the evidence available to every selector; selection cannot recover evidence absent from the candidate pool.

For the provenance stress test we mask the encoder-visible descriptor uiu_i after resolving the canonical key gig_i:

Descriptor masking0%25%50%100%
BasisDPP-RAG EM70.2068.7066.2062.40

The 7.8-point decline measures sensitivity to descriptor removal after resolution (Appendix Table 12). Resolved keys remain fixed, so the hard-ID kernel is unchanged. The inference-time treatment of unresolved provenance is described in Q4.

Q4. How are unresolved sources handled in the inference-time basis kernel?

Unresolved passages share the same basis-encoder input, [UNK_BASIS], while retaining distinct bookkeeping keys; different unresolved keys therefore do not imply distinct evidence bases in the basis kernel. Two unresolved passages have coinciding basis embeddings, so Kijb=1K^b_{ij}=1 and their pairwise similarity in the factorized kernel reduces to KijcK^c_{ij}. If every descriptor is unknown, the kernel reduces to L=WKcW+εIL=WK^cW+\varepsilon I and selection falls back to claim similarity and relevance. The same treatment holds during training: pairs containing an UNRESOLVED label are masked from Lbasis\mathcal{L}_{\mathrm{basis}}, so missing metadata never contributes a false negative asserting that two passages are independent. Consequently, genuine corroboration between unresolved passages expressing the same claim may be penalized as repetition. We now state this explicitly as a limitation.

5. Critical Weakness 1: what do the results establish about evidence dependence?

The added results support the learned basis kernel over the evaluated ID-based alternative; they do not establish that it generalizes independently of source identities.

The hard-ID comparison retains the learned claim kernel and quality weights while replacing only the basis relation with an explicit ID-based one, and BasisDPP-RAG outperforms it in all three settings reported in Q2, including natural retrieval. It does not, however, distinguish generalizable dependence modeling from all forms of reliance on source identities. Although the basis head encodes the readable descriptor uiu_i rather than the bookkeeping key gig_i, the descriptors contain source identifiers: canonical URLs for AVeriTeC, trial IDs for NLI4CT, and paper IDs for SciFact (Appendix Table 22). The source-pair audit likewise does not separately establish the accuracy of the learned kernel or rule out memorization of descriptor strings. We therefore interpret evidence independence relative to the resolved basis, as specified in Appendix E.3: the results support the utility of this representation for evidence selection, while recovery of hidden shared origins and generalization to unseen sources remain unestablished.

6. Critical Weakness 2: do the gains depend on the controlled candidate pools?

The gains are not confined to the original controlled pools: they persist after removing constructed descendants and under natural retrieval without gold-evidence protection.

You are right that the controlled pools deliberately expose the redundancy problem the method targets, and we now report how often that construction is present: constructed descendants appear in 43.2% of controlled-test queries and account for 12.8% of candidates. Removing them while retaining gold-evidence protection leaves a 2.70-point advantage over hard-ID DPP. Natural retrieval removes both the inserted descendants and gold-evidence protection; the advantage remains 2.47 points at n=12n=12 and persists across the pool sizes reported in Q3.

As noted in Appendix B.6, the controlled and natural settings differ in both pool construction and size, so their absolute EM difference does not isolate the effect of gold-evidence protection. Retrieval coverage also remains incomplete, limiting the evidence available to every selector. The supported conclusion is a selection advantage across the evaluated retrieval settings, rather than a demonstrated improvement in first-stage retrieval or provenance resolution.

Reviewer_vjZqOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper identifies a limitation that redundant passages may be selected from retrieved results, which further leads to inaccurate results. To solve it, the paper proposes BasisDPP-RAG, which combines claim and evidence-basis similarity, and passage relevance to select evidence.

Critical strengths

Clear illustrations that make it easy to catch the problem and motivation.

Critical weaknesses

The paper lacks a direct analysis on if the similarity penalty raises the selection rank of unrelated evidence and potentially causes irrelevant passages to be used in claim verification. A report on how often unrelated candidates are selected just because of the diversity penalty could address this concern.

The methodology section contains many equations without clearly showing the computational process. E.g. Section 4.2 is difficult to follow with a lot of text notations such as "Norm()" instead of an explicit definition.

Questions for authors

Since the datasets are used for selector training (as in Appendix B.5 with train/dev/test splits shown in Table 7), could you provide details about how the baselines were implemented?

That is, are the comparisons between BasisDPP-RAG and the baseline methods fair by applying the same training data and tuning settings to baselines and BasisDPP-RAG?

How does BasisDPP-RAG handle situations if unrelated evidence is selected because other candidate passages are judged to be highly redundant?

11 comments

Thank you for your careful review. We begin with the two critical weaknesses, which prompted new experiments, then answer your questions and state what the added controls establish about the contribution.

1. Does the similarity penalty promote unrelated evidence, and what happens when it does? (Critical Weakness 1; Question 3)

We added a direct audit on naturally retrieved pools (Section 6.6; Appendix Table 13). Supervised Top-kk selects 65 irrelevant passages among 2,700 (2.41%); BasisDPP-RAG selects 73 among 2,700 (2.70%). Among the 800 passages selected by our method but not by Top-kk, 36 are irrelevant (4.50%). In 8 of 1,000 audited queries (0.80%), Top-kk answers correctly while our method answers incorrectly and its newly selected evidence includes an irrelevant passage. The measured cost of diversity is therefore real but small: approximately 0.3 percentage points in the overall irrelevant-selection rate. Over the same natural pools, our method averages 70.20% EM against 63.80% for shared-score Top-kk.

On the mechanism, there is no guaranteed post-selection repair. Relevance enters only as a soft quality weight, wi=exp⁡(βri/2)w_i=\exp(\beta r_i/2), while the determinant rewards nonredundant evidence, so an irrelevant passage can be selected when its novelty outweighs its quality disadvantage. The shared generator prompt instructs the model to ignore unrelated passages, but that instruction is not a guarantee. We have corrected Appendix E.3 and Figure 8 accordingly: they now describe relevance as counterbalancing this incentive rather than categorically rejecting such passages.

Two limits of the audit, which we now state in the paper rather than leave implicit. The two selectors have equal selection counts only in aggregate, so Table 13 is no longer described as per-query cardinality-matched; Appendix B.6 specifies the diagnostic that would control this properly — selecting the feasible size-mqm_q subset maximizing ∑i∈Sri\sum_{i\in S} r_i under the same token budget — and states that Table 13 does not report it. The audit also records co-occurrence, so it does not establish that diversity caused those eight errors. We therefore claim a favorable accuracy–relevance trade-off in the tested setting, not immunity to irrelevant evidence.

2. How were the baselines implemented, and is the comparison fair? (Questions 1 and 2)

The direct selector controls provide the tightly matched comparison your question asks for. They reuse the same relevance scores rir_i from the jointly trained BasisDPP-RAG model, together with identical candidate pools, passage and token limits, generator, prompt, and decoding. Supervised Top-kk ranks by rir_i; MMR combines rir_i with passage cosine similarity; semantic DPP uses a single passage-similarity kernel with the same quality transform; rule deduplication keeps the highest-relevance item per resolved basis and fills remaining slots by relevance; hard-ID DPP retains the learned claim kernel and quality weights while replacing the learned basis kernel with a smoothed ID-equality kernel, Kijb=η+(1−η)1[gi=gj]K^b_{ij}=\eta+(1-\eta)\mathbf{1}[g_i=g_j]. Control-specific parameters are selected on development data (Appendix B.6, Table 8). With Llama-2-13B-Chat:

SelectorControlled EMΔNatural EM (n=12n{=}12)Δ
Supervised Top-kk76.47—63.80—
MMR77.47+1.0065.40+1.60
Semantic DPP77.87+1.4065.90+2.10
Rule deduplication78.87+2.4066.60+2.80
Hard-ID DPP80.50+4.0367.73+3.93
BasisDPP-RAG84.84+8.3770.20+6.40

Our own hyperparameters are selected on development data and then fixed across all three benchmarks (Table 7); the evaluation partition is never used for tuning (Appendix C.3).

For the advanced-RAG framework rows, we do not claim equalized training. They are controlled adaptations sharing candidate pools and a frozen generator, but their method-specific training and mechanisms are not thereby equalized. The revision relabels them as contextual comparisons rather than a selection-only test, and states explicitly that the original Standard RAG row is not supervised Top-kk using our trained relevance scores. Accordingly, the 4.34- and 2.47-point margins over hard-ID DPP test the learned basis representation under a shared quality signal; they do not establish identical training across framework baselines, nor that every gain is due solely to the DPP objective.

3. Making the computational process explicit (Critical Weakness 2)

Section 4.2 was the specific problem you identified, and we revised it rather than only pointing to Algorithm 1. We now define Norm⁡(v)=v/∥v∥2\operatorname{Norm}(v)=v/\lVert v\rVert_2 for v≠0v\neq0, with the zero vector mapped to zero. We state what the representations do at inference: two unresolved descriptors receive identical basis embeddings, so their mutual basis similarity is one and their pairwise dependence is governed by claim similarity; distinct bookkeeping keys do not establish independent origins; and a passage with several underlying origins still receives one basis descriptor, since the model does not assign sources to individual spans. The section now closes with the operational sequence in order — extract cic_i, resolve bib_i, predict rir_i and the two embeddings, form the factorized kernel, then select a budget-feasible set. Concretely, the RBF claim and basis kernels are multiplied element-wise and quality-weighted to give L=W(Kc⊙Kb)W+εIL=W(K^c\odot K^b)W+\varepsilon I; greedy marginal log-determinant selection then returns at most three passages within the token budget, and only their original text reaches the shared frozen generator.

4. What is new relative to generic diversity (Informativeness)

Table 2 above separates the contribution by what each selector is permitted to know, under one shared relevance signal. Generic diversity recovers little: MMR and semantic DPP gain 1.0–1.4 points on controlled pools and 1.6–2.1 on natural pools. Making provenance explicit recovers more: rule deduplication and hard-ID DPP gain 2.4–4.0 and 2.8–3.9. The learned Claim–Basis factorization gains 8.4 and 6.4, with the same ordering in both settings. The 4.34-point margin over hard-ID DPP is thus the part attributable to learning when agreement reflects repeated support, rather than reading dependence off an identifier.

Table 5 shows why claim similarity alone is insufficient: removing basis modeling leaves claim-level selection in place but raises NLI4CT selected-basis redundancy from 7.89% to 38.83% and lowers IBR/IBP from 97.85%/90.67% to 80.15%/71.28%. Theorem 1 states the corresponding pairwise property — for fixed quality weights and claim similarity, the pair determinant increases as basis similarity falls, so repetition and corroboration are separated by a quantity a claim-only kernel cannot express. The contribution is therefore not generic DPP diversity but a learned, query-invariant dependence coordinate that makes that distinction computable under a context budget. The controls you requested are also what isolates it.

Reviewer_vjZqOpenReview ↗

Response to author rebuttal

Thanks for the reviewer's feedback. It partially addressed my concerns, but I feel the answer is still a bit superficial in general and overall concerns to the paper still exist. I will maintain my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.