acceptodds
Withdrawn from ICLR 2027

Mapping the Set-Function Geometry of Evidence Utility in Retrieval-Augmented Generation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Evidence selection depends on the structure of generator utility, the search strategy, and the evaluation target. We introduce Evidence Utility Geometry, an exhaustive subset-intervention audit that measures all three within a fixed evidence interface. Across three multi-hop question-answering datasets, three generator families, and controlled versus support-blind six-passage pools, all 18 dataset–model–construction cells exhibit material non-monotonicity and increasing returns. Negative marginals occur in 27.3–37.9% of tests and positive discrete Hessians in 35.5–42.9%. Exact regret decomposition separates the price of filling a budget from positive same-cardinality search loss, including in an eight-passage extension. A complete generation audit evaluates all 64 subsets for 90 existing questions, two models, and both constructions. Generated F1 also exhibits negative marginals and positive Hessians. Likelihood–F1 rank correlations of 0.55 and 0.49 coexist with different optima: at budget three, exact likelihood selection has F1 regret of 0.184 and 0.175 against the exact F1 optimum, and its paired F1 contrasts with fixed likelihood-greedy span zero. Cardinality-standardized comparisons reveal how weighting subset sizes changes cross-size measurements. EUG connects exhaustive structural measurement to exact selection losses and target alignment, identifying where evidence-selection losses arise.

Primary area:
foundation or frontier models, including LLMs
Keywords:
retrieval-augmented generation, evidence selection, set-function geometry, greedy optimization, target alignment

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_5quUOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduces Evidence Utility Geometry, an exhaustive analysis of document-subset utility in RAG. It also compares answer likelihood with generated-answer quality, showing that their optimal evidence subsets can differ. However, the analysis and presentation need substantial development. It does not look like a version that is ready to be reviewed. These results are not yet developed into a sufficiently clear and compelling scientific narrative.

Critical strengths
  1. The exhaustive evaluation of all subsets within small evidence pools provides exact reference points for distinguishing budget-enforcement loss, offering a useful diagnostic perspective on RAG evidence selection.
Critical weaknesses
  • The manuscript reads more like an automated experimental report than a well-developed research argument. It presents numerous violation rates, regret decompositions, and statistical comparisons, but provides limited synthesis explaining what new understanding emerges from these measurements.
  • Similarly, the mismatch between likelihood and F1 is quantified, but the paper does not adequately characterize the cases driving this mismatch or derive concrete guidance for choosing an evaluation target.
  • Substantial revision is needed to make this a good version for review.
Questions for authors

N/A

Reviewer_Xi5UOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a complete subset intervention framework called Evidence Utility Geometry (EUG), which is used to analyze the evidence selection in retrieval-enhanced generation. This framework measures the non-monotonicity of the generator's utility and the improvement in returns as the input quantity increases by evaluating all evidence subsets within a fixed candidate pool, where the utility is defined by the log-likelihood of the centralized reference-answer pair. The experiments cover three multi-hop question-answering datasets, three generators, and two ways of constructing six-segment text pools, plus an eight-segment text study. Through precise regret decomposition, the cost of filling the evidence budget is separated from the same-base search loss. The complete generation answer auditing conducted on 90 questions further examines the F1 value and complete matching situations, indicating that the likelihood and answer quality may be correlated, while still prioritizing different evidence subsets. Additionally, the paper also shows that the comparison results between different pool sizes depend on the weight allocation of the subset base.

Critical strengths
  1. Exhaustive subset evaluation enables precise diagnosis. By enumerating every evidence subset, the paper obtains exact cardinality-constrained optima and fully observed finite differences, avoiding the limitations of analyzing only the subsets visited by a particular selector.

  2. The framework separates multiple, complementary sources of selection failure. It distinguishes the cardinality price, the same-cardinality search loss, and the likelihood–reward mismatch, thereby explaining why evidence selection can underperform.

  3. The experimental and statistical design is meticulous. The study covers three datasets, three generators, and both support-complete and support-blind candidate constructions. Question-clustered bootstrap appropriately preserves dependence across subsets and models, and the nested cohort structure and the absence of a defined reward correlation are documented explicitly.

Critical weaknesses
  1. The explanatory power of several measures is not fully established. The main correlation analysis links the prevalence of positive Hessians to total fixed‑cardinality greedy regret, which aggregates the cardinality price and the same‑cardinality search loss. It does not establish whether this measure specifically explains same‑cardinality search loss, or whether it adds information beyond utility magnitude and negative‑marginal intensity at the question level.

  2. The null‑evidence interface is not controlled precisely enough. The empty subset admits different representations of null evidence, and the minimum‑cardinality layer exhibits a particularly high violation rate. Although the interface effect is explicitly incorporated into the estimator, the current controls do not adequately distinguish evidence interaction from formatting changes or from shifts in the model’s response strategy under the null condition.

  3. The incremental contribution and cross‑scale explanation require stronger support. The regret decomposition is a useful accounting identity rather than a substantive theoretical result; consequently, the contribution rests mainly on the joint empirical audits. In addition, cardinality standardization only determines how subset size is weighted; it does not fix candidate composition, and thus cannot reveal how increasing the number of candidates would change the interaction structure.

Questions for authors
  1. After controlling for question‑level utility range and negative‑marginal intensity, is the prevalence of positive Hessians still associated with same‑cardinality search loss? Does it provide explanatory value beyond these simpler statistics?

  2. All set‑function measurements are conditional on the specific prompt, serialization and null‑evidence formatting pipeline. While the authors acknowledge this interface dependency, it remains unclear to what extent reported non‑monotonicity and complementarity reflect intrinsic properties of evidence content versus measurement artifacts induced by the evaluation setup.

  3. The core observation that likelihood‑optimal evidence subsets diverge from F1‑optimal subsets exhibits high sensitivity to question cohort and model composition. Instead of merely expanding experimental scale, could the authors identify underlying driving factors that modulate the magnitude of this misalignment, such as question difficulty, multi‑hop reasoning complexity or noise level within the retrieved passage pool?

Reviewer_Mi3HOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces Evidence Utility Geometry (EUG), an exhaustive audit of evidence-subset utility under a fixed RAG interface. The primary study evaluates centered, teacher-forced canonical-answer log-likelihood over all 64 subsets of six-passage pools for 540 questions, three generators, and two local candidate constructions. An eight-passage likelihood extension covers 180 existing questions and two generators in Track A. Complete generated-answer evaluation additionally covers 90 existing questions, two generators, and both tracks. The study measures non-monotonicity and increasing returns, separates fixed-greedy regret into cardinality price and same-cardinality search loss, and examines alignment between likelihood and generated-answer rewards. It finds recurring structural violations and positive F1 regret for exact likelihood selection despite positive likelihood–F1 rank association. Cardinality-standardized comparisons further show that cross-size measurements depend on the weighting of subset sizes.

Critical strengths
  1. Complete enumeration provides exact within-interface references. Evaluating every subset makes finite differences, constrained optima, and selection regrets directly measurable without approximating the subset search. Material likelihood violations recur in all 18 primary cells at the primary threshold, and pooled threshold-sensitivity analyses support their persistence across reporting thresholds (Tables 8 and 10).

  2. The separation of cardinality price and search loss is useful and correct. Equation 4 distinguishes the best achievable cost of requiring exactly b passages from the additional loss of choosing a suboptimal set of that size. The separate stopped-greedy analysis and pair-escape diagnostics show why singleton stopping can miss improving combinations (Table 4; Appendix D). The contribution here is the empirical diagnostic use of the decomposition.

  3. The complete generated-reward audit materially strengthens the study. Scoring all subsets permits direct comparison with the exact F1 optimum and distinguishes landscape correlation from optimal-value alignment. The additional F1-greedy diagnostics show residual selection regret even when greedy decisions use the reward itself, although that regret includes both cardinality and combination effects (Tables 5–6 and 14).

  4. The statistical and interpretive reporting is careful. The paper preserves question-level dependence, retains constant reward cubes in sign-rate denominators, accounts for undefined correlations, and explains nested populations. The cardinality-standardized analysis also usefully demonstrates that different lattice weightings answer different questions, without claiming to isolate a causal effect of pool size (Appendices A.3, E, F, and G).

Critical weaknesses
  1. The population-level geometry–regret analysis is less specific than the loss decomposition allows. Table 9 relates positive-Hessian mass to total fixed-greedy regret, which includes both cardinality price and same-cardinality search loss. Reporting associations with these components separately would clarify which selection problem the statistic helps diagnose. The existing correlations are valid descriptive associations, but they do not resolve this distinction.

  2. The quantitative target-alignment gap has not been reassessed under alias-aware likelihood. The primary likelihood objective scores one canonical answer, while F1 accepts official aliases. Appendix H.2 reports alias sensitivity for likelihood geometry and likelihood regret, but not the generated-F1 regret of the alias-aware likelihood optimum. The reported F1 gaps of 0.184 and 0.175 remain valid for the declared canonical objective; the unresolved issue is how much their magnitude depends on that reference formulation.

  3. The policy-level reward comparisons have limited precision and model coverage. Complete reward lattices cover 90 questions and two generators. Appendix F transparently shows that the exact-U-versus-fixed-greedy point estimate changes with model and cohort composition. Model-specific paired intervals would help assess the consistency of the aggregate comparison. The present results establish neither equivalence nor fixed-greedy superiority, and the paper appropriately avoids those claims.

  4. Decision-level transfer beyond the canonical interface remains limited. The experiments use small benchmark-local pools, fixed passage representations, and short-answer lexical rewards. The order controls principally identify matched local finite differences: only 27 of 1,080 noncanonical conditions identify exact at-most-budget-three optima (Appendix H.1). These controls therefore provide more direct evidence about local order sensitivity than about stability of exact selection losses across serializations. The authors acknowledge this restriction; it limits the reach of the diagnostic conclusions rather than invalidating the canonical-interface results.

Questions for authors
  1. Component-specific geometry associations: Using the existing likelihood cubes, can you report the association of positive-Hessian mass separately with cardinality price, same-cardinality search loss, and stopped-greedy regret? A sensitivity accounting for per-question utility range would also help distinguish variation in utility amplitude from other aspects of the measured geometry.

  2. Alias sensitivity of target alignment: On appropriately matched questions with alternative official aliases, how do the F1 regret of exact likelihood selection and the principal policy contrasts change under the alias-max objective from Appendix H.2? This would clarify the robustness of the quantitative mismatch across reference formulations.

  3. Precision of model-specific policy contrasts: For the models evaluated in the respective parent and complete-generation populations, can you report exact-U-minus-fixed-greedy F1 contrasts with paired uncertainty? Table 15 already explains the aggregate composition descriptively; the requested intervals would clarify the precision and consistency of the model-specific comparisons.

  4. Clarification of retained support information: For the support-recall comparison in Section 5.2, how does passage-label recall relate to supporting content retained after the 256-token cap? Where supporting-sentence annotations are available, a descriptive retention statistic would help interpret this secondary diagnostic. This question does not presume that truncation removed the relevant evidence or affected the principal geometry results.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.