acceptodds
Under review as a conference paper at ICLR 2027

Shapley-Guided Set-Wise Reranking for Retrieval-Augmented Generation

Abstract

Retrieval-augmented generation (RAG) relies on reranking to select a small set of documents for a downstream LLM under a limited context budget. Existing rerankers either score documents individually or rank candidates jointly, but lack a principled way to model both the utility of a document subset and each document's contribution to that utility. We formulate RAG reranking as a cooperative game, where documents are players and a coalition's payoff measures its usefulness for the query. We introduce a set-wise valuer to model this coalition utility and show that a local desirability condition on the valuer is sufficient to induce the desired document ordering under a symmetric semivalue. This result motivates a set-contrastive training objective that avoids explicitly computing attribution values during training. At inference, we instantiate the same local-comparison principle with a practical restricted regression attribution rule. Specifically, we sample small coalitions according to a truncated KernelSHAP-inspired distribution and fit an additive regression to their utilities. We further train an amortized estimator to predict these attribution scores in a single forward pass. Across five multi-hop QA benchmarks, our method outperforms strong pointwise, listwise, and setwise baselines in reranking recall and downstream answer quality in most cases, while the amortized estimator retains most of the gain at the cost of a standard pointwise reranker, yielding a speedup over sampling-based attribution.

Primary area:
foundation or frontier models, including LLMs
Keywords:
RAG reranking, Shapley, Set-wise reranking

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_jp5MOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes SHAPRank, a framework for RAG reranking that learns a utility function over document subsets and ranks documents using restricted, Shapley-inspired attribution. It combines set-wise contrastive training with sampling-based attribution and an amortized estimator that predicts document scores in a single forward pass. I like the idea overall, but the experimental analysis is not sufficient, I will raise the score if the author can provide a good revision in the rebuttal.

Critical strengths
  • Efficiently estimating document utility in the presence of cross-document interactions is an interesting and practically relevant direction for RAG.

  • Combining sampling-based attribution with an amortized estimator is a sensible design. The reported results suggest that much of the benefit of repeated coalition evaluation can be retained with a single forward pass.I think it is an interesting idea for inspiring the community to design related methods.

Critical weaknesses
  • Scalability remains insufficiently established. The method reduces online computation under the evaluated setting, but experiments primarily use 30 candidates and inference-time coalitions containing at most four documents. Larger candidate pools and longer evidence contexts are relevant to many RAG applications. It is unclear how the required sampling budget, latency, and memory usage change with 100–200 candidates, or whether restricting coalition size misses higher-order interactions.

  • The analysis provides limited insight into why the method works. The ablations, including comparisons with greedy and beam-search selection, establish that the components contribute empirically. However, several conceptual questions remain unresolved. The valuer is trained using supporting-document labels rather than downstream answer quality, so it is unclear whether it learns answer utility or primarily aggregates relevance signals. Moreover, allocating credit among documents and selecting the best subset under a context budget are different objectives.

  • The positioning against closely related work should be strengthened. Source Attribution in Retrieval-Augmented Generation [1] studies document-level Shapley attribution, approximation costs, and redundancy, complementarity, and synergy. RepoShapley [2] uses Shapley-style contributions to filter context in code RAG and distills offline selection decisions into a model. Shapley Context Pruning [3] is especially close in combining a learned set-value function with Shapley-based context ranking and pruning. The paper should clarify its methodological differences and discuss appropriate comparisons, accounting for the publication timing of concurrent work.

  • The experimental discussion is overly descriptive. Much of the discussion restates performance differences and attributes improvements to complementary evidence without directly testing that explanation. The author should polish their writing a lot (it feels like using AI to help write a lot).

[1] Nematov et al. Source Attribution in Retrieval-Augmented Generation. 2025. [2] Huo et al. RepoShapley: Shapley-Enhanced Context Filtering for Repository-Level Code Completion. Findings of ACL, 2026. [3] Chen et al. Shapley Context Pruning: A Cooperative Game Perspective for Context Reranking and Pruning. 2026.

Questions for authors
  • How should the sampling budget scale with candidate-pool size, and what interactions are lost when inference-time coalitions contain at most four documents?

  • What evidence shows that the learned valuer captures downstream answer utility beyond supporting-document relevance?

  • Under a fixed valuer and computation budget, does the proposed weighting outperform uniform coalition sampling or alternative semivalue weights?

7 comments
AuthorsOpenReview ↗

Response to reviewer jp5M –part 1

Thank you for your encouraging review and for recognizing the potential of our approach to inspire future reranker designs. We also appreciate your thoughtful questions on scalability, learned utility, and related work, which have helped us sharpen our contributions.


Q1. Scalability, sampling-budget scaling, and higher-order interactions

Covers Weakness 1 and Question 1.

Response:

Candidate-pool size and coalition size affect different aspects of scalability: the former determines how many documents must receive reliable scores, while the latter determines which document combinations the valuer observes. We separate these factors in the following evaluations.

1. Larger candidate pools: fixed versus scaled sampling budgets. We evaluate our SHAPRank-S on the top 30, 50, 100, and 200 documents from the same first-stage retriever on MuSiQue, using the same queries and frozen valuer. We fix the maximum coalition size at B=4B=4 and compare T=200T=200 with a scaled budget T(n)=⌈200n/30⌉T(n)=\lceil200n/30\rceil. Increasing the sampling budget in proportion to the candidate count keeps the average number of evaluations per document approximately unchanged. All results below use a query batch size of 1 on the Musique dataset.

Candidates nnSampling budgetCoalitions TTR@5R@10Latency (s/query)
30Fixed20066.8174.094.87
50Fixed20067.2275.414.87
50Scaled33467.8975.928.10
100Fixed20067.1275.234.87
100Scaled66768.3676.6412.8
200Fixed20066.6174.014.87
200Scaled133468.4576.7125.39

Since the memory usage is related to batch size and coalition size, varying candidate pools do not directly affect the memory usage. In our experiments, it requires ~30GB GPU memory usage.

The results indicate that larger candidate pools improve retrieval performance when accompanied by a larger sampling budget: R@5/R@10 increases from 66.81/74.09 with 30 candidates, to 68.45/76.71 with 200. At a fixed budget of 200 coalitions, however, the gains level off: expanding from 50 to 200 candidates slightly reduces both metrics. Increasing the budget at 200 candidates significantly improves metrics over its fixed-budget sampling counterpart.

This pattern is consistent with reduced sampling coverage per document: when each coalition contains at most four documents, a fixed number of sampled coalitions gives each document fewer opportunities to be evaluated as the pool grows. Scaling the sampling budget helps realize the benefit of the larger candidate pool. This comes with additional computation: latency rises from 4.87 seconds at 30 candidates to 12.8 seconds at 100 candidates with the scaled budget. The results therefore support increasing the sampling budget alongside the candidate pool, with a measurable trade-off between recall and inference cost.

2. Larger coalitions: extending the sensitivity analysis in Figure 4. Figure 4 already compares maximum coalition sizes B=2,3,4B=2,3,4 across sampling budgets, with results averaged over all five benchmarks. Expanding the range from 1–2 to 1–3 documents improves recall, while extending it to 1–4 provides little additional improvement. This analysis motivates the default small-coalition setting.

To examine whether larger coalitions provide further benefits, we extend the comparison to B=6,8,10B=6,8,10, keeping the existing trained valuer fixed and using the same candidate pools (n=30n=30) and sampling budget (T=200T=200). No additional valuer training is performed. The table below reports average R@5 and R@10 across the five benchmarks, together with inference latency. Since the valuer was trained on sets of 1–4 documents, the additional settings also test its generalization to larger input sets.

Maximum coalition size BBR@5R@10Latency (s/query)
4 (default reference)81.0286.334.88
681.0586.436.24
881.1586.498.19
1080.8086.009.59

Increasing the maximum coalition size beyond four yields only small gains: at B=8B=8, R@5 and R@10 improve by just 0.13 and 0.16 percentage points over the default, while latency rises from 4.88 to 8.19 seconds per query. Together with Figure 4, this suggests that much of the useful interaction information for these benchmarks can already be captured through small document groups. Increasing the limit to B=10B=10 slightly reduces performance. One plausible explanation is that evaluating larger combinations makes it harder for the fixed valuer to distinguish their relative usefulness of these much larger doc sets.

Theoretically, increasing coalition sizes will also decrease the weight. Hence, these results favor B=4B=4 as a practical balance. Larger coalitions provide limited additional benefit in our evaluated setting, while requiring longer inputs to be scored.

AuthorsOpenReview ↗

Response to reviewer jp5M –part 2

Q2. Learned utility, document attribution, and subset selection

Covers Weakness 2 and Question 2.

Response:

Thank you for raising these important distinctions. We clarify that our valuer learns evidence-set utility as a proxy for supporting-evidence quality, rather than directly predicting downstream answer quality. This is because final answer quality also depends on the generator and prompting strategy. Our goal is to identify useful evidence combinations and translate their utility into effective document rankings.

1. Does the valuer learn meaningful set utility?

Figure 6 evaluates the valuer's ability to distinguish evidence sets with different numbers of supporting documents. The set-wise-trained valuer achieves an average satisfaction rate of 86.6%, compared with 77.8% for the original model and 77.1% for point-wise training, across five benchmarks.

Furthermore, our new controlled experiment (Reviewer hivt, Q1) compares complementary and redundant evidence sets with identical set sizes and supporting-document counts. The set-wise-trained valuer prefers complementary evidence in 92.8% of pairs on MuSiQue and 93.5% on MultiHopRAG, substantially outperforming point-wise training (74.8% and 75.8%). These results demonstrate that the valuer captures evidence composition beyond supporting-document counts. Additional downstream F1 improvements under a fixed generator further support the practical value of the learned utility.

2. How do set-wise learning and attribution contribute to ranking?

Figure 3 isolates their contributions using controlled ablations with matched backbones and training conditions:

Valuer trainingWithout attribution: Average R@5 / R@10With attribution: Average R@5 / R@10
Point-wise76.20 / 80.8077.71 / 83.41
Set-wise76.42 / 81.1381.02 / 86.33

Set-wise training alone improves average R@5 by only 0.22 points, whereas combining set-wise training with attribution yields a 4.60-point gain over set-wise training alone. This demonstrates their synergy: set-wise training captures evidence interactions, while attribution converts this information into effective document rankings.

3. How does attribution relate to subset selection?

Attribution and subset selection are distinct but connected. SHAPRank-S estimates document scores from sampled coalition utilities by fitting an additive approximation. Selecting the top-mm documents maximizes this approximated set utility under a fixed document budget, rather than directly optimizing the original, potentially non-additive utility. This provides an efficient approximation to subset selection. Empirically, SHAPRank-S achieves 81.02 average R@5, outperforming greedy selection (78.37) and beam search (79.76) using the same valuer.

Together, these results support our central contribution: learning evidence-set utility and leveraging attribution to guide effective document ranking, without requiring generator-specific answer-quality supervision.


Q3. Relation to Source Attribution in RAG, RepoShapley, and Shapley Context Pruning

Covers Weakness 3.

Response:

Thank you for highlighting these related studies. Source Attribution in RAG explains document influence on a particular generator's response, while RepoShapley distills generator-verified context-filtering decisions for code completion. SHAPRank learns evidence rankings from supporting-document supervision, without requiring generator-based utility labels or verified filtering decisions.

SCP is the closest related work, but differs substantially in training and inference. SCP trains a Deep Sets network on estimated Shapley scores and uses permutation-based Monte Carlo attribution. In contrast, our theoretical connection between local set comparisons and semivalue ordering enables set-wise training without precomputed attribution scores. Our valuer jointly encodes the query and document set, rather than aggregating independently encoded sentences. At inference, SHAPRank-S uses restricted coalition regression, while SHAPRank-A predicts document scores in a single pass.

Our core contribution is a training principle for RAG reranking: learning evidence-set utility and using it to guide document ranking. By connecting set-wise training with attribution-based ranking, we provide both sampling-based and efficient amortized inference, without requiring generator-based utility labels. We will clarify these distinctions in the revised related-work section.

AuthorsOpenReview ↗

Response to reviewer jp5M – part 3

Q4. Experimental discussion and evidence for the explanations

Covers Weakness 4.

Response:

Thank you for your feedback. In response to your concern, we have carefully rewritten Sections 6.1–6.4 to provide clearer explanations and stronger experimental evidence. Specifically:

  • Section 6.1: Clarify performance comparisons and add the set-wise trained baseline without attribution.
  • Section 6.2: Better explain the downstream benefits of improved evidence ranking.
  • Section 6.3: Clarify inference efficiency and quality–cost trade-offs.
  • Section 6.4: Strengthen the analysis of the synergy between set-wise training and attribution, supported by additional controlled ablations on coalition-utility supervision and weighting.

We also add experiments on complementary versus redundant evidence to further validate the learned set utility (see Reviewer hivt, Q1). We hope these revisions provide a clearer, more evidence-driven explanation of our experimental findings and adequately address your concern.


Q5. Does the weighting beat uniform sampling or other semivalues?

Covers Question 3.

Response:

Thank you for suggesting this controlled comparison. As reported in our response to Reviewer hivt, Q1, we compare restricted KernelSHAP weighting with uniform sampling over coalition sizes, holding the valuer, candidate pools, maximum coalition size, and evaluation budget fixed. Restricted KernelSHAP achieves 81.02 average R@5, compared with 79.43 for the uniform-size baseline, a gain of 1.59 percentage points, with improvements on all five datasets. The full per-dataset results are provided in that response.

Since both configurations use the same frozen valuer, this comparison isolates the effect of weighting beyond coalition aggregation. The consistent improvements across all five datasets support our theoretically motivated weighting scheme, which emphasizes smaller coalitions while still capturing multi-document interactions. We will include this comparison in the revised ablation study.


We sincerely hope that our clarifications and additional experiments have adequately addressed your concerns. Thank you again for your valuable feedback and consideration. Please do not hesitate to let us know if you have any further questions or concerns.

Reviewer_FLEWOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SHAPRank, a set-wise reranking framework for retrieval-augmented generation (RAG) based on cooperative game theory. It learns document-subset utilities using a set-wise contrastive objective and estimates individual document contributions through Shapley-inspired attribution. Two variants are introduced: SHAPRank-S, which uses coalition sampling and regression-based attribution, and SHAPRank-A, which directly predicts attribution scores for efficient inference. Experiments on five multi-hop QA benchmarks demonstrate improvements in average retrieval recall and downstream answer quality.

Critical strengths
  1. Modeling document reranking as a cooperative game provides a principled approach to capturing interactions and complementary evidence across documents.

  2. The connection between local subset utility comparisons and semivalue ordering provides justification for the proposed set-wise contrastive objective.

  3. The paper introduces an innovative attribution-based reranking approach that captures document contributions to subset utility, with efficient amortized inference.

  4. The paper evaluates multiple benchmarks and compares against point-wise, list-wise, and set-wise rerankers, including controlled ablation studies.

Critical weaknesses
  1. Although the paper includes a no-attribution ablation, reporting per-dataset results for the set-wise fine-tuned model without attribution in the main comparison table would better highlight the contribution of the proposed attribution method.

  2. SHAPRank does not consistently outperform existing rerankers across all benchmarks. For example, SHAPRank-A underperforms the original Qwen3-Reranker on MultiHopRAG. Further analysis of these performance differences would strengthen the evaluation.

  3. The experiments only use Qwen3-Reranker as the backbone for SHARPRank method. Evaluating SHAPRank with other reranker architectures or applying attribution to an off-the-shelf model without set-wise fine-tuning would better demonstrate its generalizability.

  4. Although the paper examines inference-time coalition sizes, it does not systematically investigate how different subset sizes during set-wise fine-tuning affect reranking performance.

  5. The paper states ModernBERT is agnostic to SHARPRank-A framework. Experiments with alternative encoder architectures would provide stronger evidence for this claimed model-agnostic property and the generalizability of amortized attribution. Such experiment would help establish whether the effectiveness and efficiency of amortized attribution generalize across encoder choice.

Questions for authors
  1. Could the authors include the set-wise fine-tuned model without attribution in the main comparison table?

  2. Why does SHAPRank underperform some baselines on MultiHopRAG?

  3. Can SHAPRank-S and SHAPRank-A be applied to pretrained rerankers without set-wise fine-tuning or to different model architectures?

  4. What is the effect of subset size during set-wise finetuning?

5 comments
AuthorsOpenReview ↗

Response to reviewer FLEW – part 1

Thank you for your constructive review and valuable suggestions on experimental baselines, generalization, and architectural flexibility. Your feedback has helped us strengthen the experimental validation of SHAPRank.


Q1. Add the set-wise fine-tuned model without attribution to the main table

Covers Weakness 1 and Question 1.

Response:

Thank you for this helpful suggestion. We agree that including the set-wise fine-tuned model without attribution in the main table would make the contribution of our attribution mechanism easier to assess. These per-dataset results are already available in the appendix, and we will move the relevant row into the main table to make this comparison more visible. This baseline uses the same set-wise fine-tuned valuer as SHAPRank-S, but ranks documents by singleton utility vθ(q,{di})v_\theta(q,\{d_i\}), allowing us to examine the additional benefit of attribution while holding the trained valuer fixed.

DatasetSet-wise FT, no attribution: R@5 / R@10SHAPRank-S: R@5 / R@10
MuSiQue63.20 / 69.4466.81 / 74.09
HotpotQA90.05 / 93.6092.95 / 95.60
2WikiMQA70.13 / 72.8573.20 / 77.00
MultiHopRAG81.02 / 89.9384.01 / 92.89
MorehopQA77.70 / 79.8588.15 / 92.10
Average76.42 / 81.1381.02 / 86.33

SHAPRank-S improves over this baseline on all five datasets, with average gains of 4.60 percentage points in R@5 and 5.20 percentage points in R@10. Because both methods use the same trained valuer, these gains demonstrate the additional benefit of computing document contributions across sampled coalitions, beyond the benefit of set-wise fine-tuning alone. In particular, singleton scoring evaluates each document in isolation, whereas SHAPRank-S accounts for its marginal contribution in the presence of other documents. The results therefore support the value of using the learned set utility for attribution when constructing multi-document contexts.

Q2. Why does performance drop on MultiHopRAG?

Covers Weakness 2 and Question 2.

Response:

Thank you for raising this point. MultiHopRAG differs from our training dataset in how questions specify the required evidence. Its news-based queries include comparisons between named sources and temporal relations between reports. MuSiQue, on which both our valuer and amortized estimator are trained, emphasizes dependent reasoning steps across Wikipedia passages. Thus, transferring to MultiHopRAG requires matching evidence to source and time constraints as well as combining relevant facts.

In our experiments, each MultiHopRAG candidate contains only its title and body, without separate source or publication-date fields. Consider an illustrative question asking us to compare Source A's September report with Source B's November report about the same company. An October report from Source B may discuss the same company and event, but it cannot replace the November report specified in the question. We suspect that our models, trained only on MuSiQue, may give high scores to combinations of related facts with multi-step reasoning patterns while overlooking these source or date requirements needed by MultiHopRAG, particularly when this information is not explicit in the title and body. Besides, SHAPRank-A instead predicts document scores directly using a model trained on MuSiQue, and its predictions may be more sensitive to this change in evidence requirements of the MultiHopRAG dataset.

AuthorsOpenReview ↗

Response to reviewer FLEW – part 2

Q3. Can the method use an off-the-shelf reranker or a different valuer architecture?

Covers Weakness 3 and Question 3.

Response:

According to your concern, we add an ablation experiment on the off-the-shelf reranker model and another reranker from a different family.

Each result cell reports R@5. All methods use the same candidate pools; SHAPRank-S uses B=4B=4 and T=200T=200.

Valuer backboneSet-wise fine-tuningScoring methodMuSiQueHotpotQA2WikiMQAMultiHopRAGMorehopQAAverage
Qwen3-Reranker-0.6BNoSingleton scoring60.2291.0072.5884.4086.7578.99
Qwen3-Reranker-0.6BNoSHAPRank-S61.3091.5672.8884.4887.2079.48
Qwen3-Reranker-0.6BYesSingleton scoring63.2090.0570.1381.0277.7076.42
Qwen3-Reranker-0.6BYesSHAPRank-S66.8192.9573.2084.0188.1581.02
BGE-v2-m3NoSingleton scoring61.7792.7571.2082.2079.7077.52
BGE-v2-m3NoSHAPRank-S62.1092.8972.1083.3579.8978.06
BGE-v2-m3YesSingleton scoring61.9091.2070.2481.2379.3476.78
BGE-v2-m3YesSHAPRank-S67.1393.5773.1283.9182.3080.01

From the results, we can find that the SHAPRank-S could also lift the perfomance of the off-the-shelf reranker Qwen3-Reranker-0.6B, from 78.99 to 79.48 on average. If the model is set-wise finetuned, the improvement will be larger, from 76.42 to 81.02. This comparison verifies the effectiveness of the attribution mechanism of SHAPRank on off-the-shelf models. Moreover, for BGE-v2-m3 from another family, we observed similar improvement for off-the-shelf version (from 77.52 to 78.06). For the set-wise trained version of BGE-v2-m3, the improvement is larger (76.78 to 80.01), which is expected.

However, we emphasize that the SHAPRank requires the reranker model to be able to assign accurate relative scores for document sets. It is expected to further finetune an off-the-shelf model to enhance this ability. Hence, the improvement on set-wise-finetuned models is more significant than that on a non-finetuned counterpart.


Q4. What is the effect of subset size during training?

Covers Weakness 4 and Question 4.

Response:

Thank you for highlighting the effect of subset size during training. To address this question directly, we vary the training background-size range while keeping the inference settings fixed, separating the effect of training set sizes from that of inference-time coalition sizes. The table below reports R@5 across the five benchmarks.

Training background sizes ∣S∣\lvert S\rvertTraining set sizes ∣S∪{d±}∣\lvert S\cup\{d^\pm\}\rvertMuSiQueHotpotQA2WikiMQAMultiHopRAGMorehopQAAverage
0 (point-wise trained)162.7689.2070.5382.3383.7577.71
0–11–263.3990.4771.3682.9184.8578.60
0–21–364.8391.3572.7283.5886.8279.86
0–3 (default)1–466.8192.9573.2084.0188.1581.02
0–41–566.8292.9673.2383.9888.2081.04

The rows “0 (point-wise trained)” and “0–3 (default)” reuse results already reported in the paper: the former is the point-wise-trained model with sampling-based attribution, since an empty training background reduces each compared set to a single document. The latter is SHAPRank-S from the main table, whose default training set sizes of 1–4 correspond to background sizes of 0–3.

As the training set-size range expands from singletons to sets of up to four documents, average R@5 increases from 77.71 to 78.60, 79.86, 81.02, and 81.04. This trend suggests that exposure to larger training sets helps the valuer learn to compare the relative utility of document sets in the presence of other evidence; sampling-based attribution then aggregates these context-dependent utility differences into more effective document rankings. Extending the training range to five documents yields almost saturated average performance (81.02 vs 81.04), indicating diminishing returns beyond the default range in this experiment. These results support training on multi-document sets, while suggesting that small sets are sufficient to capture much of the benefit under our evaluation settings.

AuthorsOpenReview ↗

Response to reviewer FLEW – part 3

Q5. Does amortized attribution generalize beyond ModernBERT?

Covers Weakness 5 and the estimator part of Question 3.

Response:

Thank you for suggesting an empirical test with an alternative encoder. To address this concern directly, we use GTE-en-MLM-large (~435M parameters), which is close in size to our default ModernBERT encoder and supports an 8192-token context window. Both estimators use the same frozen valuer and coalition supervision. The dataset columns below report R@5, and latency is measured in seconds per query.

Estimator encoderMuSiQueHotpotQA2WikiMQAMultiHopRAGMorehopQAAverage R@5Latency (s/query)
ModernBERT (default)67.0793.1073.7581.3991.3581.330.68
GTE-en-MLM-large (~435M)67.2193.0673.5781.5191.4081.350.76

GTE-en-MLM-large achieves almost the same average R@5 as ModernBERT (81.33 vs 81.35, rounded to two decimal places), with per-dataset differences of at most 0.18 percentage points. Its inference latency increases from 0.68 to 0.76 seconds per query. These results indicate that the effectiveness of the amortized estimator is retained with this alternative encoder, while the precise inference cost depends on the backbone. We will include this comparison in the revised paper to support encoder flexibility empirically.


We sincerely hope that our clarifications and additional experiments have adequately addressed your concerns. Thank you again for your valuable feedback and consideration. Please do not hesitate to let us know if you have any further questions or concerns.

Reviewer_hivtOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SHAPRANK formulates RAG reranking as a cooperative game, learning document-set utilities and assigning document-level credit. Its key contribution links local utility comparisons to attribution-based ordering, enabling contrastive training without explicitly computing attribution scores. Experiments on five multi-hop QA benchmarks show improved retrieval and answer quality, with amortization substantially reducing latency.

Critical strengths
  1. Matched training and architecture controls isolate the benefits of SHAPRANK-A’s coalition-based aggregation, which improves average R@5 and downstream F1 over Qwen3-Reranker-0.6B.

  2. Evaluation on four additional benchmarks supports cross-dataset generalization, while SHAPRANK-A reduces per-query latency from 4.87 to 0.68 seconds compared with sampling-based attribution.

Critical weaknesses
  1. The experiments show that coalition-based aggregation helps, but do not establish whether the gains come from modeling complementarity and redundancy or from the specific Shapley-inspired weighting.
  2. Training contexts of sizes 1–4 differ from the sizes 0–3 in Theorem 3. The paper should also specify the context sizes used for evaluation, how contexts are truncated, and how random seeds vary across runs.
Questions for authors
  1. What is the exact distribution used to sample training contexts SS, and what probability is assigned to each context size? Does the range {1,2,3,4}\{1,2,3,4\} in Appendix B.4 refer to ∣S∣|S| or ∣S∪{d±}∣|S\cup\{d^\pm\}|? If it refers to ∣S∣|S|, why are empty contexts excluded, given that they account for 25% of the population regression-gap weighting under the default inference setting?
  2. Have you compared the proposed method with simpler coalition-size weighting schemes using the same frozen valuer and evaluation budget? When the number of supporting documents is controlled, is there evidence that the learned utility function distinguishes complementary from redundant evidence?
3 comments
AuthorsOpenReview ↗

Response to reviewer hivt

Thank you for your insightful review and thoughtful questions about the sources of our performance gains and the details of set-wise training. Your feedback has helped us clarify our methodology and strengthen our empirical analysis.


Q1. What causes the gains: set aggregation, evidence interactions, or Shapley-inspired weights?

Covers Weakness 1 and Question 2.

Response:

Thank you for raising this important question. Our main finding is that the performance gains arise from the synergy between set-wise training and attribution: set-wise training enables the valuer to capture evidence-set utility, while attribution translates this learned utility into effective document rankings. Our controlled ablations show that their combination achieves 81.02 average R@5, compared with 76.42 for set-wise training without attribution and 77.71 for attribution applied to a point-wise-trained valuer. We further investigate the roles of coalition weighting and evidence complementarity below.

1. Effect of coalition weighting.

To isolate the contribution of our restricted KernelSHAP weighting, we compare it with uniform sampling over coalition sizes, keeping the frozen valuer, candidate pools, maximum coalition size, and evaluation budget fixed. All results report R@5.

Coalition weighting / samplingMuSiQueHotpotQA2WikiMQAMultiHopRAGMorehopQAAverage R@5
Restricted KernelSHAP (ours)66.8192.9573.2084.0188.1581.02
Uniform over coalition sizes65.3290.8872.3483.2985.3479.43

Restricted KernelSHAP weighting consistently outperforms uniform sampling across all five datasets, improving average R@5 by 1.59 percentage points. Since both methods aggregate coalition utilities from the same frozen valuer, this comparison demonstrates the additional benefit of our weighting scheme beyond coalition aggregation itself.

2. Does the valuer capture evidence complementarity?

To examine whether set-wise training learns meaningful evidence-set utility rather than simply counting supporting documents, we construct matched pairs of complementary and redundant evidence sets, holding both set size and supporting-document count fixed. Complementary sets cover different reasoning steps, whereas redundant sets repeat evidence for the same step.

We measure the percentage of pairs where the valuer assigns a higher score to the complementary set. Separately, we evaluate downstream answer F1 using the same candidate pools, fixed generator, selected-document budget, and generation settings. F1 gains are reported relative to the original Qwen3-Reranker.

DatasetValuerNumber of matched pairsValuer preference (%)Answer F1 gain over Original Qwen3 (pp)
MuSiQueOriginal Qwen3-Reranker500073.9%+0.0
MuSiQuePoint-wise fine-tuned Qwen3-Reranker500074.8%+0.5
MuSiQueSet-wise fine-tuned Qwen3-Reranker500092.8%+3.1
Out-of-domain benchmark (MultiHopRAG)Original Qwen3-Reranker500075.2%+0.0
Out-of-domain benchmark (MultiHopRAG)Point-wise fine-tuned Qwen3-Reranker500075.8%+0.2
Out-of-domain benchmark (MultiHopRAG)Set-wise fine-tuned Qwen3-Reranker500093.5%+2.6

The set-wise-trained valuer strongly favors complementary evidence, achieving preference rates of 92.8% on MuSiQue and 93.5% on MultiHopRAG, substantially higher than its point-wise-trained counterpart. Since supporting-document counts are identical, these results indicate that the valuer learns to distinguish evidence composition rather than merely accumulating relevance. The downstream F1 gains (+3.1 and +2.6 points) further support the practical usefulness of the learned set utility.

Together, these results reinforce our central finding: set-wise training captures the utility of evidence combinations, while attribution effectively leverages this utility for document ranking. The restricted KernelSHAP weighting further improves how these coalition utilities are translated into ranking scores.

AuthorsOpenReview ↗

Response to reviewer hivt

Q2. Training context sizes, empty contexts, truncation, and random seeds

Covers Weakness 2 and Question 1.

Response:

Thank you for pointing out the notation. The training sizes 1–4 refer to the complete sets evaluated by the valuer, S∪{d+}S\cup\{d^+\} and S∪{d−}S\cup\{d^-\}, rather than the background SS alone. Each set contains one compared document, so the shared background has size 0–3. Theorem 3 applies to a general coalition-size limit BB, with background sizes 0,…,B−10,\ldots,B-1; the range 0–3 follows from our default setting B=4B=4, rather than a fixed restriction of the theorem, and matches our training backgrounds.

In particular, a training set of size one corresponds to S=∅S=\varnothing. Training then directly compares v(q,{d+})v(q,\{d^+\}) with v(q,{d−})v(q,\{d^-\}). Empty backgrounds are therefore included, rather than excluded. We agree that their total mass in the default regression-gap distribution is 25%; their actual training sampling probability is uniform sampling over coalition sizes.

Appendix B.4 incorrectly assigns the range 1–4 to ∣S∣|S|. We will correct it to ∣S∪{d±}∣∈{1,2,3,4}|S\cup\{d^\pm\}|\in\{1,2,3,4\}, or equivalently ∣S∣∈{0,1,2,3}|S|\in\{0,1,2,3\}. This corrects the description of the existing training setup, but it does not change the method or require retraining.

During evaluation, SHAPRank-S evaluates sampled coalitions containing 1–4 documents. SHAPRank-A takes all 30 candidate documents as input, and the downstream generator receives the top five reranked documents. We retain the full text of the documents without truncation.

For SHAPRank-S, we keep the trained valuer fixed across the three runs and vary only the inference coalition-sampling seed, using seeds 42, 43, and 44. The reported mean and standard deviation reflect variation from inference-time coalition sampling in the method.


We sincerely hope that our clarifications and additional experiments have adequately addressed your concerns. Thank you again for your valuable feedback and consideration. Please do not hesitate to let us know if you have any further questions or concerns.

AuthorsOpenReview ↗

Gentle Follow-up on Our Rebuttal

Dear Reviewer,

Thank you again for your valuable comments and suggestions.

We would like to kindly follow up on our previous rebuttal. We would greatly appreciate it if you could take a moment to review our responses when you have time.

Please let us know if you have any further questions or concerns. We would be happy to provide additional clarification.

Thank you for your time and consideration.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.