- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2(+1)
- Soundness
- 3(+1)
Read review
Paper summary
This paper explains how can we give sources fair credit for generative source engines. Shapley takes much time to recalculate score for every subset using LLM, so Maxshapley, splits sources into key points and measure scores for each key point. Using HotPotQA, MuSiQUE, and MS MARCO experiments, MaxShapley agrees better with human relevance labels than FullShapley, while using about 5% of FullShapley's tokens.
Critical strengths
- Demonstration webpage intuitively shows how the framework is, and why this work is important. Also it is much more effective than just showing the code, that this framework is really working.
- Algorithm 1 is succinct and valid.
- Even though they are in the appendix, Fig. 4 and Fig. 5 helped me understand the main idea. I recommend this to be in the main explanation.
Critical weaknesses
- In the abstract, Maxshapley is a special case of Shapely value. So it is read to be a cheaper proxy of Fullshapley. However after reading, it seems to be different from Fullshapley and better clarity is needed.
- At section 1.1, LLM-as-a-judge utility function (Kendall-tau>0.5) and strong alignment with human annotations (Jaccard index>0.85) is written as achievement but they do not hold for HotpotQA and MS-Marco.
- Section 4.1.1 relevance alignment derivation seems to be not used anywhere else in the experiment.
- Minor point : MS-Marco and MS Marco is both used as a terminology.
- Fig. 3 token consumption is not clear which consumption we are referring to. Also, error bar in the figure is not sure if it is calculated from 3 runs or 100 runs.
- Fig. 6 or Fig. 7 seems to be wrong (maybe Fig. 7). They look same.
- “See more details in Appendix 4.3.” seems to be referring section 4.3 itself.
Questions for authors
- Function U that is used in Maxshapley is the score that LLM gave, so it really depends on the model. Is this true? Also, whole framework seems to be guaranteeing fair attribution when U is given. Can we guarantee U really reflects realistic contribution?
- It seems that if Maxshapley gives reward for wrong answers. In section 4.2, it says “On MS-MARCO, we observe a degradation in the quality of attribution across all Shapley attribution methods”, but wrong answer’s attribution is not analyzed. Shouldn’t the answer be dependent to the valid reasoning from reference?
- If two relevant sources are paid 99% and 1%, the relevance-consistency loss is still zero, just like a 50/50 split. Does Theorem 4.3 tell us anything about how fairly credit is split among relevant sources, which is the main reason to use Shapley values?
- Is reward calculation speed important, having less latency? Is the framework detrimental to be offline?
- There are multiple mentioning about customization and also mentioned as a contribution in section 1.1, but I did not find relevant evidence for it. For example, all experiments use single scoring prompt, uniform weights, and also it seems that prompt needs model specific finetuning. How should we consider customization and how can this be strength of the paper?
5 comments
Response to Reviewer Qf4R – Part 2 of 3
Q3: Interpretation and empirical evaluation of relevance alignment.
“Section 4.1.1 relevance alignment derivation seems to be not used anywhere else in the experiment.”
“If two relevant sources are paid 99% and 1%, the relevance-consistency loss is still zero, just like a 50/50 split. Does Theorem 4.3 tell us anything about how fairly credit is split among relevant sources, which is the main reason to use Shapley values?”
Response: First, we note that this is not a problem of Shapley values or Theorem 4.3, but of Jaccard@K. Section 4.1.1 and Theorem 4.3 therein establish the economic implications of Jaccard@K for payment allocation, with empirical evaluation in Appendix F. It is Jaccard itself that does not distinguish the 99%/1% and 50%/50% splits, because it measures relevance ranking rather than how credit is divided among relevant sources.
We still view Jaccard score as a useful metric because it measures whether the highest-paid sources are the relevant ones. However, to address your point, we have added another metric to our experiments of Jain’s fairness index (Jain et al., 1984) to evaluate payment balance among relevant sources. In your example, the index is approximately 0.5101 for the 99%/1% split, compared with 1 for the 50%/50% split, i.e., it differentiates the example you gave in a way that Jaccard cannot. We report this additional evaluation below. MaxShapley achieves the highest mean Jain’s fairness index on HotPotQA and MuSiQUE and ranks second (to Full Shapley) on MS-MARCO. We have included this evaluation in Appendix J.1 of the revised manuscript.
Jain’s fairness index among benchmark-relevant sources
| Method | HotPotQA | MuSiQUE | MS-MARCO |
|---|---|---|---|
| MaxShapley | 0.85 ± 0.02 | 0.85 ± 0.02 | 0.72 ± 0.04 |
| FullShapley | 0.75 ± 0.02 | 0.73 ± 0.02 | 0.83 ± 0.02 |
| LOO | 0.65 ± 0.02 | 0.78 ± 0.03 | 0.51 ± 0.04 |
| MCU | 0.58 ± 0.02 | 0.58 ± 0.02 | 0.57 ± 0.04 |
| MCA | 0.72 ± 0.02 | 0.72 ± 0.02 | 0.63 ± 0.04 |
| KernelSHAP | 0.58 ± 0.02 | 0.58 ± 0.02 | 0.57 ± 0.04 |
Each query is evaluated in three runs. Values are means ± standard errors across query averages. Higher values indicate more equal payments among benchmark-relevant sources. The highest mean among the displayed methods is shown in bold.
Q4: Practical benefits of customization.
“There are multiple mentioning about customization and also mentioned as a contribution in section 1.1, but I did not find relevant evidence for it. For example, all experiments use single scoring prompt, uniform weights, and also it seems that prompt needs model specific finetuning. How should we consider customization and how can this be strength of the paper?”
Response: Although MaxShapley here is presented and evaluated in the usual context of generative search, different search or advertising systems may prioritize different criteria, such as relevance, factuality, helpfulness, or source use. MaxShapley can accommodate these criteria by customizing its generation and scoring prompts, or extending its coverage utility, without requiring model fine-tuning. In the revised manuscript, we have added two experiments demonstrating this flexibility: Relevance-MaxShapley (Appendix J.2) and Corroboration-MaxShapley (Appendix C.3),
For convenience, we copy the results from Relevance-MaxShapley here (please also see the response to Reviewer UPXf above for a listing of results for Corroboration-MaxShapley). , We consider a setting in which the platform seeks to avoid rewarding unused sources and reduce credit assigned to distracting evidence. We implement this policy through **Relevance-MaxShapley whose modified scorer assigns zero support to unused sources and downweights source–keypoint pairs judged unhelpful. This yields zero payment to unused sources and consistently increases the payment share assigned to benchmark-relevant sources across all three datasets (see the tables below), while maintaining comparable source-ranking performance: mean Jaccard and Kendall scores change by at most 0.0325 and 0.0266, respectively.
Fraction of payment assigned to used sources (%, higher is better)
| Method | HotPotQA | MuSiQUE | MS-MARCO |
|---|---|---|---|
| MaxShapley | 94.7 ± 1.0 | 97.0 ± 0.9 | 94.3 ± 0.8 |
| Relevance-MaxShapley | 100.0 ± 0.0 | 100.0 ± 0.0 | 100.0 ± 0.0 |
| FullShapley | 84.2 ± 1.7 | 75.5 ± 2.6 | 64.1 ± 1.9 |
Fraction of payment assigned to relevant sources (%, higher is better)
| Method | HotPotQA | MuSiQUE | MS-MARCO |
|---|---|---|---|
| MaxShapley | 92.1 ± 1.7 | 96.5 ± 1.5 | 83.0 ± 2.4 |
| Relevance-MaxShapley | 95.1 ± 1.4 | 97.5 ± 1.3 | 85.4 ± 2.4 |
| FullShapley | 85.4 ± 1.7 | 78.6 ± 2.2 | 57.7 ± 2.2 |
Each query is evaluated in three runs. Values are means ± standard errors across query averages. The highest mean among the displayed methods is shown in bold.
Response to Reviewer Qf4R – Part 3 of 3
Q5: Importance of computational efficiency.
“Is reward calculation speed important, having less latency? Is the framework detrimental to be offline?”
Response: Low latency matters if, for example, attribution scores are needed to select advertisements shown alongside the answer, as proposed in Appendix B. For direct payments, MaxShapley can instead run after the answer is served, using the saved query, retrieved sources, and generated answer. Offline calculation does not limit that use; avoiding repeated generation still reduces the cost of processing many queries (Figure 4 in the revised manuscript).
Q6: Experimental claims.
“At section 1.1, LLM-as-a-judge utility function (Kendall-tau>0.5) and strong alignment with human annotations (Jaccard index>0.85) is written as achievement but they do not hold for HotpotQA and MS-Marco.”
Response: Thank you for pointing out that the dataset-specific results could be clearer. We first note that our more recent MaxShapley experiments using GPT-4.1-mini report substantially higher Jaccard and Kendall scores than the GPT-4.1-nano results in the submission (see table below). We have included these results in Appendix J of the revised manuscript.
Relevance-ranking performance using GPT-4.1-mini
| Metric | Method | HotPotQA | MuSiQUE | MS-MARCO |
|---|---|---|---|---|
| Jaccard@K | MaxShapley | 0.922 ± 0.019 | 0.977 ± 0.013 | 0.759 ± 0.030 |
| Jaccard@K | FullShapley | 0.842 ± 0.026 | 0.867 ± 0.027 | 0.660 ± 0.031 |
| Kendall | MaxShapley | 0.723 ± 0.024 | 0.687 ± 0.029 | 0.393 ± 0.028 |
| Kendall | FullShapley | 1.000 ± 0.000 | 1.000 ± 0.000 | 1.000 ± 0.000 |
Each query is evaluated in three runs. Values are means ± standard errors.
We also have revised Section 1.1 to reflect the results more precisely. In particular, we have revised the statement to "LLM-as-a-judge utility function (Kendall-tau>0.5) and strong alignment with human annotations (Jaccard index>0.8) on HotPotQA and MuSiQUE". MS-MARCO is a challenging benchmark because its less curated source passages can be confusing, even for humans (Section 4.2): all methods have Jaccard <0.7, and all methods (except FullShapley’s self-comparison) have Kendall b<0.4.
Q7: Token accounting.
“Fig. 3 token consumption is not clear which consumption we are referring to. Also, error bar in the figure is not sure if it is calculated from 3 runs or 100 runs.”
Response: Token consumption denotes the total input and output tokens used per query, including answer generation and all LLM-based attribution stages. For MaxShapley, these include keypoint extraction, refinement, and support scoring; for FullShapley, they include generation and judging across source subsets. Each query is evaluated three times. The plotted values are means across queries, and the error bars represent standard errors across queries. We have clarified these definitions in the experimental setup.
Q8: Presentation improvements.
“Even though they are in the appendix, Fig. 4 and Fig. 5 helped me understand the main idea. I recommend this to be in the main explanation.”
“Minor point : MS-Marco and MS Marco is both used as a terminology.”
“Fig. 6 or Fig. 7 seems to be wrong (maybe Fig. 7). They look same.”
“‘See more details in Appendix 4.3.’ seems to be referring section 4.3 itself.”
Response: Thank you for these helpful presentation suggestions. We have brought the illustrations in Figures 4 and 5 into the main explanation of MaxShapley as Figure 3 in the revised manuscript, standardized the dataset name as MS-MARCO, replaced Figure 7’s duplicated content with the intended keypoint breakdown prompt, and corrected “Appendix 4.3” to “Appendix G.”
Response to Reviewer Qf4R – Part 1 of 3
We thank you for your thoughtful review and hope that our responses, together with new experiments demonstrating the practical benefits of customization and evaluating payment fairness using Jain’s index, address your concerns. We have included these additional results in the revised manuscript and would appreciate it if you could positively reconsider your assessment in light of these additions.
Q1: Relationship to FullShapley and validity of the utility.
“In the abstract, Maxshapley is a special case of Shapely value. So it is read to be a cheaper proxy of Fullshapley. However after reading, it seems to be different from Fullshapley and better clarity is needed.”
“Function U that is used in Maxshapley is the score that LLM gave, so it really depends on the model. Is this true? Also, whole framework seems to be guaranteeing fair attribution when U is given. Can we guarantee U really reflects realistic contribution?”
Response: Thank you, you are correct: MaxShapley computes exact Shapley values for the max-sum utility in Equation (5), whereas FullShapley computes Shapley values for the regeneration-based answer-quality utility in Equation (4). Thus, they apply the same Shapley definition to different utilities; MaxShapley is not an approximation of FullShapley’s allocation. We have clarified this distinction in the main text of the revised manuscript.
Regarding MaxShapley’s utility, yes, it depends on the LLM’s support scores. This modularity also allows us to adapt MaxShapley to different platforms’ objectives, as demonstrated in Section 4.4 of the revised manuscript. Our benchmark results also empirically suggest that our utility indeed reflects realistic contributions: MaxShapley outperforms FullShapley in Jaccard relevance ranking across all three datasets. It also achieves the highest Jain’s fairness index among benchmark-relevant sources on HotPotQA and MuSiQUE, ranking second to FullShapley on MS-MARCO. These results are also shown in our later response to your question Q3 on fairness evaluation and given in Appendix J of the revised manuscript.
Q2: Attribution when the generated answer is incorrect.
“It seems that Maxshapley gives reward for wrong answers." "Shouldn’t the answer be dependent to the valid reasoning from reference?”
Response: Regarding whether the answer should depend on valid reasoning from the references, MaxShapley focuses on attributing the generated answer rather than independently verifying each claim’s correctness. This is also an issue in online advertising, where compensation generally depends on content use rather than claim-by-claim assessment of correctness (e.g., Google AdSense). The same principle applies to our setting, except that sources are aggregated into an answer by an AI model rather than accessed individually by a user. Furthermore, many questions lack a single objectively correct answer. Comprehensive factual verification and its evaluation extend beyond this work’s focus and are therefore left to future work.