- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes SHAPRank, a framework for RAG reranking that learns a utility function over document subsets and ranks documents using restricted, Shapley-inspired attribution. It combines set-wise contrastive training with sampling-based attribution and an amortized estimator that predicts document scores in a single forward pass. I like the idea overall, but the experimental analysis is not sufficient, I will raise the score if the author can provide a good revision in the rebuttal.
Critical strengths
-
Efficiently estimating document utility in the presence of cross-document interactions is an interesting and practically relevant direction for RAG.
-
Combining sampling-based attribution with an amortized estimator is a sensible design. The reported results suggest that much of the benefit of repeated coalition evaluation can be retained with a single forward pass.I think it is an interesting idea for inspiring the community to design related methods.
Critical weaknesses
-
Scalability remains insufficiently established. The method reduces online computation under the evaluated setting, but experiments primarily use 30 candidates and inference-time coalitions containing at most four documents. Larger candidate pools and longer evidence contexts are relevant to many RAG applications. It is unclear how the required sampling budget, latency, and memory usage change with 100–200 candidates, or whether restricting coalition size misses higher-order interactions.
-
The analysis provides limited insight into why the method works. The ablations, including comparisons with greedy and beam-search selection, establish that the components contribute empirically. However, several conceptual questions remain unresolved. The valuer is trained using supporting-document labels rather than downstream answer quality, so it is unclear whether it learns answer utility or primarily aggregates relevance signals. Moreover, allocating credit among documents and selecting the best subset under a context budget are different objectives.
-
The positioning against closely related work should be strengthened. Source Attribution in Retrieval-Augmented Generation [1] studies document-level Shapley attribution, approximation costs, and redundancy, complementarity, and synergy. RepoShapley [2] uses Shapley-style contributions to filter context in code RAG and distills offline selection decisions into a model. Shapley Context Pruning [3] is especially close in combining a learned set-value function with Shapley-based context ranking and pruning. The paper should clarify its methodological differences and discuss appropriate comparisons, accounting for the publication timing of concurrent work.
-
The experimental discussion is overly descriptive. Much of the discussion restates performance differences and attributes improvements to complementary evidence without directly testing that explanation. The author should polish their writing a lot (it feels like using AI to help write a lot).
[1] Nematov et al. Source Attribution in Retrieval-Augmented Generation. 2025. [2] Huo et al. RepoShapley: Shapley-Enhanced Context Filtering for Repository-Level Code Completion. Findings of ACL, 2026. [3] Chen et al. Shapley Context Pruning: A Cooperative Game Perspective for Context Reranking and Pruning. 2026.
Questions for authors
-
How should the sampling budget scale with candidate-pool size, and what interactions are lost when inference-time coalitions contain at most four documents?
-
What evidence shows that the learned valuer captures downstream answer utility beyond supporting-document relevance?
-
Under a fixed valuer and computation budget, does the proposed weighting outperform uniform coalition sampling or alternative semivalue weights?
7 comments
Response to reviewer jp5M –part 2
Q2. Learned utility, document attribution, and subset selection
Covers Weakness 2 and Question 2.
Response:
Thank you for raising these important distinctions. We clarify that our valuer learns evidence-set utility as a proxy for supporting-evidence quality, rather than directly predicting downstream answer quality. This is because final answer quality also depends on the generator and prompting strategy. Our goal is to identify useful evidence combinations and translate their utility into effective document rankings.
1. Does the valuer learn meaningful set utility?
Figure 6 evaluates the valuer's ability to distinguish evidence sets with different numbers of supporting documents. The set-wise-trained valuer achieves an average satisfaction rate of 86.6%, compared with 77.8% for the original model and 77.1% for point-wise training, across five benchmarks.
Furthermore, our new controlled experiment (Reviewer hivt, Q1) compares complementary and redundant evidence sets with identical set sizes and supporting-document counts. The set-wise-trained valuer prefers complementary evidence in 92.8% of pairs on MuSiQue and 93.5% on MultiHopRAG, substantially outperforming point-wise training (74.8% and 75.8%). These results demonstrate that the valuer captures evidence composition beyond supporting-document counts. Additional downstream F1 improvements under a fixed generator further support the practical value of the learned utility.
2. How do set-wise learning and attribution contribute to ranking?
Figure 3 isolates their contributions using controlled ablations with matched backbones and training conditions:
| Valuer training | Without attribution: Average R@5 / R@10 | With attribution: Average R@5 / R@10 |
|---|---|---|
| Point-wise | 76.20 / 80.80 | 77.71 / 83.41 |
| Set-wise | 76.42 / 81.13 | 81.02 / 86.33 |
Set-wise training alone improves average R@5 by only 0.22 points, whereas combining set-wise training with attribution yields a 4.60-point gain over set-wise training alone. This demonstrates their synergy: set-wise training captures evidence interactions, while attribution converts this information into effective document rankings.
3. How does attribution relate to subset selection?
Attribution and subset selection are distinct but connected. SHAPRank-S estimates document scores from sampled coalition utilities by fitting an additive approximation. Selecting the top- documents maximizes this approximated set utility under a fixed document budget, rather than directly optimizing the original, potentially non-additive utility. This provides an efficient approximation to subset selection. Empirically, SHAPRank-S achieves 81.02 average R@5, outperforming greedy selection (78.37) and beam search (79.76) using the same valuer.
Together, these results support our central contribution: learning evidence-set utility and leveraging attribution to guide effective document ranking, without requiring generator-specific answer-quality supervision.
Q3. Relation to Source Attribution in RAG, RepoShapley, and Shapley Context Pruning
Covers Weakness 3.
Response:
Thank you for highlighting these related studies. Source Attribution in RAG explains document influence on a particular generator's response, while RepoShapley distills generator-verified context-filtering decisions for code completion. SHAPRank learns evidence rankings from supporting-document supervision, without requiring generator-based utility labels or verified filtering decisions.
SCP is the closest related work, but differs substantially in training and inference. SCP trains a Deep Sets network on estimated Shapley scores and uses permutation-based Monte Carlo attribution. In contrast, our theoretical connection between local set comparisons and semivalue ordering enables set-wise training without precomputed attribution scores. Our valuer jointly encodes the query and document set, rather than aggregating independently encoded sentences. At inference, SHAPRank-S uses restricted coalition regression, while SHAPRank-A predicts document scores in a single pass.
Our core contribution is a training principle for RAG reranking: learning evidence-set utility and using it to guide document ranking. By connecting set-wise training with attribution-based ranking, we provide both sampling-based and efficient amortized inference, without requiring generator-based utility labels. We will clarify these distinctions in the revised related-work section.
Response to reviewer jp5M – part 3
Q4. Experimental discussion and evidence for the explanations
Covers Weakness 4.
Response:
Thank you for your feedback. In response to your concern, we have carefully rewritten Sections 6.1–6.4 to provide clearer explanations and stronger experimental evidence. Specifically:
- Section 6.1: Clarify performance comparisons and add the set-wise trained baseline without attribution.
- Section 6.2: Better explain the downstream benefits of improved evidence ranking.
- Section 6.3: Clarify inference efficiency and quality–cost trade-offs.
- Section 6.4: Strengthen the analysis of the synergy between set-wise training and attribution, supported by additional controlled ablations on coalition-utility supervision and weighting.
We also add experiments on complementary versus redundant evidence to further validate the learned set utility (see Reviewer hivt, Q1). We hope these revisions provide a clearer, more evidence-driven explanation of our experimental findings and adequately address your concern.
Q5. Does the weighting beat uniform sampling or other semivalues?
Covers Question 3.
Response:
Thank you for suggesting this controlled comparison. As reported in our response to Reviewer hivt, Q1, we compare restricted KernelSHAP weighting with uniform sampling over coalition sizes, holding the valuer, candidate pools, maximum coalition size, and evaluation budget fixed. Restricted KernelSHAP achieves 81.02 average R@5, compared with 79.43 for the uniform-size baseline, a gain of 1.59 percentage points, with improvements on all five datasets. The full per-dataset results are provided in that response.
Since both configurations use the same frozen valuer, this comparison isolates the effect of weighting beyond coalition aggregation. The consistent improvements across all five datasets support our theoretically motivated weighting scheme, which emphasizes smaller coalitions while still capturing multi-document interactions. We will include this comparison in the revised ablation study.
We sincerely hope that our clarifications and additional experiments have adequately addressed your concerns. Thank you again for your valuable feedback and consideration. Please do not hesitate to let us know if you have any further questions or concerns.
Response to reviewer jp5M –part 1
Thank you for your encouraging review and for recognizing the potential of our approach to inspire future reranker designs. We also appreciate your thoughtful questions on scalability, learned utility, and related work, which have helped us sharpen our contributions.
Q1. Scalability, sampling-budget scaling, and higher-order interactions
Covers Weakness 1 and Question 1.
Response:
Candidate-pool size and coalition size affect different aspects of scalability: the former determines how many documents must receive reliable scores, while the latter determines which document combinations the valuer observes. We separate these factors in the following evaluations.
1. Larger candidate pools: fixed versus scaled sampling budgets. We evaluate our SHAPRank-S on the top 30, 50, 100, and 200 documents from the same first-stage retriever on MuSiQue, using the same queries and frozen valuer. We fix the maximum coalition size at B=4 and compare T=200 with a scaled budget T(n)=⌈200n/30⌉. Increasing the sampling budget in proportion to the candidate count keeps the average number of evaluations per document approximately unchanged. All results below use a query batch size of 1 on the Musique dataset.
Since the memory usage is related to batch size and coalition size, varying candidate pools do not directly affect the memory usage. In our experiments, it requires ~30GB GPU memory usage.
The results indicate that larger candidate pools improve retrieval performance when accompanied by a larger sampling budget: R@5/R@10 increases from 66.81/74.09 with 30 candidates, to 68.45/76.71 with 200. At a fixed budget of 200 coalitions, however, the gains level off: expanding from 50 to 200 candidates slightly reduces both metrics. Increasing the budget at 200 candidates significantly improves metrics over its fixed-budget sampling counterpart.
This pattern is consistent with reduced sampling coverage per document: when each coalition contains at most four documents, a fixed number of sampled coalitions gives each document fewer opportunities to be evaluated as the pool grows. Scaling the sampling budget helps realize the benefit of the larger candidate pool. This comes with additional computation: latency rises from 4.87 seconds at 30 candidates to 12.8 seconds at 100 candidates with the scaled budget. The results therefore support increasing the sampling budget alongside the candidate pool, with a measurable trade-off between recall and inference cost.
2. Larger coalitions: extending the sensitivity analysis in Figure 4. Figure 4 already compares maximum coalition sizes B=2,3,4 across sampling budgets, with results averaged over all five benchmarks. Expanding the range from 1–2 to 1–3 documents improves recall, while extending it to 1–4 provides little additional improvement. This analysis motivates the default small-coalition setting.
To examine whether larger coalitions provide further benefits, we extend the comparison to B=6,8,10, keeping the existing trained valuer fixed and using the same candidate pools (n=30) and sampling budget (T=200). No additional valuer training is performed. The table below reports average R@5 and R@10 across the five benchmarks, together with inference latency. Since the valuer was trained on sets of 1–4 documents, the additional settings also test its generalization to larger input sets.
Increasing the maximum coalition size beyond four yields only small gains: at B=8, R@5 and R@10 improve by just 0.13 and 0.16 percentage points over the default, while latency rises from 4.88 to 8.19 seconds per query. Together with Figure 4, this suggests that much of the useful interaction information for these benchmarks can already be captured through small document groups. Increasing the limit to B=10 slightly reduces performance. One plausible explanation is that evaluating larger combinations makes it harder for the fixed valuer to distinguish their relative usefulness of these much larger doc sets.
Theoretically, increasing coalition sizes will also decrease the weight. Hence, these results favor B=4 as a practical balance. Larger coalitions provide limited additional benefit in our evaluated setting, while requiring longer inputs to be scored.