acceptodds
Under review as a conference paper at ICLR 2027

MaxShapley: Towards Fair Context Attribution for Generative Search

Abstract

Generative search engines based on large language models (LLMs) are replacing traditional search, fundamentally changing how information providers are compensated. To sustain this ecosystem, we need fair mechanisms to attribute and compensate content providers based on their contributions to generated answers. We introduce MaxShapley, an efficient algorithm for fair credit attribution in generative search pipelines that retrieve external sources before generation. MaxShapley is a special case of the celebrated Shapley value; it leverages a decomposable max-sum utility function to compute attributions with polynomial-time computation in the number of documents, as opposed to the exponential worst-case cost of Shapley values. We evaluate MaxShapley on three multi-hop QA datasets (HotPotQA, MuSiQUE, MS MARCO); MaxShapley achieves comparable attribution quality to an LLM-based full Shapley computation, while consuming a fraction of its tokens—for instance, it gives up to a 9x reduction in resource consumption over prior state-of-the-art methods at the same attribution accuracy. We release open-source code and recalibrated datasets. A demonstration is available at https://fair-search.com.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
generative search, fair context attribution, LLM search, incentives, content providers

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Qf4ROpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2(+1)
Soundness
3(+1)
Read review
Paper summary

This paper explains how can we give sources fair credit for generative source engines. Shapley takes much time to recalculate score for every subset using LLM, so Maxshapley, splits sources into key points and measure scores for each key point. Using HotPotQA, MuSiQUE, and MS MARCO experiments, MaxShapley agrees better with human relevance labels than FullShapley, while using about 5% of FullShapley's tokens.

Critical strengths
  1. Demonstration webpage intuitively shows how the framework is, and why this work is important. Also it is much more effective than just showing the code, that this framework is really working.
  2. Algorithm 1 is succinct and valid.
  3. Even though they are in the appendix, Fig. 4 and Fig. 5 helped me understand the main idea. I recommend this to be in the main explanation.
Critical weaknesses
  1. In the abstract, Maxshapley is a special case of Shapely value. So it is read to be a cheaper proxy of Fullshapley. However after reading, it seems to be different from Fullshapley and better clarity is needed.
  2. At section 1.1, LLM-as-a-judge utility function (Kendall-tau>0.5) and strong alignment with human annotations (Jaccard index>0.85) is written as achievement but they do not hold for HotpotQA and MS-Marco.
  3. Section 4.1.1 relevance alignment derivation seems to be not used anywhere else in the experiment.
  4. Minor point : MS-Marco and MS Marco is both used as a terminology.
  5. Fig. 3 token consumption is not clear which consumption we are referring to. Also, error bar in the figure is not sure if it is calculated from 3 runs or 100 runs.
  6. Fig. 6 or Fig. 7 seems to be wrong (maybe Fig. 7). They look same.
  7. “See more details in Appendix 4.3.” seems to be referring section 4.3 itself.
Questions for authors
  1. Function U that is used in Maxshapley is the score that LLM gave, so it really depends on the model. Is this true? Also, whole framework seems to be guaranteeing fair attribution when U is given. Can we guarantee U really reflects realistic contribution?
  2. It seems that if Maxshapley gives reward for wrong answers. In section 4.2, it says “On MS-MARCO, we observe a degradation in the quality of attribution across all Shapley attribution methods”, but wrong answer’s attribution is not analyzed. Shouldn’t the answer be dependent to the valid reasoning from reference?
  3. If two relevant sources are paid 99% and 1%, the relevance-consistency loss is still zero, just like a 50/50 split. Does Theorem 4.3 tell us anything about how fairly credit is split among relevant sources, which is the main reason to use Shapley values?
  4. Is reward calculation speed important, having less latency? Is the framework detrimental to be offline?
  5. There are multiple mentioning about customization and also mentioned as a contribution in section 1.1, but I did not find relevant evidence for it. For example, all experiments use single scoring prompt, uniform weights, and also it seems that prompt needs model specific finetuning. How should we consider customization and how can this be strength of the paper?
5 comments
AuthorsOpenReview ↗

Response to Reviewer Qf4R – Part 1 of 3

We thank you for your thoughtful review and hope that our responses, together with new experiments demonstrating the practical benefits of customization and evaluating payment fairness using Jain’s index, address your concerns. We have included these additional results in the revised manuscript and would appreciate it if you could positively reconsider your assessment in light of these additions.

Q1: Relationship to FullShapley and validity of the utility.

“In the abstract, Maxshapley is a special case of Shapely value. So it is read to be a cheaper proxy of Fullshapley. However after reading, it seems to be different from Fullshapley and better clarity is needed.”

“Function U that is used in Maxshapley is the score that LLM gave, so it really depends on the model. Is this true? Also, whole framework seems to be guaranteeing fair attribution when U is given. Can we guarantee U really reflects realistic contribution?”

Response: Thank you, you are correct: MaxShapley computes exact Shapley values for the max-sum utility in Equation (5), whereas FullShapley computes Shapley values for the regeneration-based answer-quality utility in Equation (4). Thus, they apply the same Shapley definition to different utilities; MaxShapley is not an approximation of FullShapley’s allocation. We have clarified this distinction in the main text of the revised manuscript.

Regarding MaxShapley’s utility, yes, it depends on the LLM’s support scores. This modularity also allows us to adapt MaxShapley to different platforms’ objectives, as demonstrated in Section 4.4 of the revised manuscript. Our benchmark results also empirically suggest that our utility indeed reflects realistic contributions: MaxShapley outperforms FullShapley in Jaccard relevance ranking across all three datasets. It also achieves the highest Jain’s fairness index among benchmark-relevant sources on HotPotQA and MuSiQUE, ranking second to FullShapley on MS-MARCO. These results are also shown in our later response to your question Q3 on fairness evaluation and given in Appendix J of the revised manuscript.

Q2: Attribution when the generated answer is incorrect.

“It seems that Maxshapley gives reward for wrong answers." "Shouldn’t the answer be dependent to the valid reasoning from reference?”

Response: Regarding whether the answer should depend on valid reasoning from the references, MaxShapley focuses on attributing the generated answer rather than independently verifying each claim’s correctness. This is also an issue in online advertising, where compensation generally depends on content use rather than claim-by-claim assessment of correctness (e.g., Google AdSense). The same principle applies to our setting, except that sources are aggregated into an answer by an AI model rather than accessed individually by a user. Furthermore, many questions lack a single objectively correct answer. Comprehensive factual verification and its evaluation extend beyond this work’s focus and are therefore left to future work.

AuthorsOpenReview ↗

Response to Reviewer Qf4R – Part 2 of 3

Q3: Interpretation and empirical evaluation of relevance alignment.

“Section 4.1.1 relevance alignment derivation seems to be not used anywhere else in the experiment.”

“If two relevant sources are paid 99% and 1%, the relevance-consistency loss is still zero, just like a 50/50 split. Does Theorem 4.3 tell us anything about how fairly credit is split among relevant sources, which is the main reason to use Shapley values?”

Response: First, we note that this is not a problem of Shapley values or Theorem 4.3, but of Jaccard@K. Section 4.1.1 and Theorem 4.3 therein establish the economic implications of Jaccard@K for payment allocation, with empirical evaluation in Appendix F. It is Jaccard itself that does not distinguish the 99%/1% and 50%/50% splits, because it measures relevance ranking rather than how credit is divided among relevant sources.

We still view Jaccard score as a useful metric because it measures whether the highest-paid sources are the relevant ones. However, to address your point, we have added another metric to our experiments of Jain’s fairness index (Jain et al., 1984) to evaluate payment balance among relevant sources. In your example, the index is approximately 0.5101 for the 99%/1% split, compared with 1 for the 50%/50% split, i.e., it differentiates the example you gave in a way that Jaccard cannot. We report this additional evaluation below. MaxShapley achieves the highest mean Jain’s fairness index on HotPotQA and MuSiQUE and ranks second (to Full Shapley) on MS-MARCO. We have included this evaluation in Appendix J.1 of the revised manuscript.

Jain’s fairness index among benchmark-relevant sources

MethodHotPotQAMuSiQUEMS-MARCO
MaxShapley0.85 ± 0.020.85 ± 0.020.72 ± 0.04
FullShapley0.75 ± 0.020.73 ± 0.020.83 ± 0.02
LOO0.65 ± 0.020.78 ± 0.030.51 ± 0.04
MCU0.58 ± 0.020.58 ± 0.020.57 ± 0.04
MCA0.72 ± 0.020.72 ± 0.020.63 ± 0.04
KernelSHAP0.58 ± 0.020.58 ± 0.020.57 ± 0.04

Each query is evaluated in three runs. Values are means ± standard errors across query averages. Higher values indicate more equal payments among benchmark-relevant sources. The highest mean among the displayed methods is shown in bold.

Q4: Practical benefits of customization.

“There are multiple mentioning about customization and also mentioned as a contribution in section 1.1, but I did not find relevant evidence for it. For example, all experiments use single scoring prompt, uniform weights, and also it seems that prompt needs model specific finetuning. How should we consider customization and how can this be strength of the paper?”

Response: Although MaxShapley here is presented and evaluated in the usual context of generative search, different search or advertising systems may prioritize different criteria, such as relevance, factuality, helpfulness, or source use. MaxShapley can accommodate these criteria by customizing its generation and scoring prompts, or extending its coverage utility, without requiring model fine-tuning. In the revised manuscript, we have added two experiments demonstrating this flexibility: Relevance-MaxShapley (Appendix J.2) and Corroboration-MaxShapley (Appendix C.3),

For convenience, we copy the results from Relevance-MaxShapley here (please also see the response to Reviewer UPXf above for a listing of results for Corroboration-MaxShapley). , We consider a setting in which the platform seeks to avoid rewarding unused sources and reduce credit assigned to distracting evidence. We implement this policy through **Relevance-MaxShapley whose modified scorer assigns zero support to unused sources and downweights source–keypoint pairs judged unhelpful. This yields zero payment to unused sources and consistently increases the payment share assigned to benchmark-relevant sources across all three datasets (see the tables below), while maintaining comparable source-ranking performance: mean Jaccard and Kendall scores change by at most 0.0325 and 0.0266, respectively.

Fraction of payment assigned to used sources (%, higher is better)

MethodHotPotQAMuSiQUEMS-MARCO
MaxShapley94.7 ± 1.097.0 ± 0.994.3 ± 0.8
Relevance-MaxShapley100.0 ± 0.0100.0 ± 0.0100.0 ± 0.0
FullShapley84.2 ± 1.775.5 ± 2.664.1 ± 1.9

Fraction of payment assigned to relevant sources (%, higher is better)

MethodHotPotQAMuSiQUEMS-MARCO
MaxShapley92.1 ± 1.796.5 ± 1.583.0 ± 2.4
Relevance-MaxShapley95.1 ± 1.497.5 ± 1.385.4 ± 2.4
FullShapley85.4 ± 1.778.6 ± 2.257.7 ± 2.2

Each query is evaluated in three runs. Values are means ± standard errors across query averages. The highest mean among the displayed methods is shown in bold.

AuthorsOpenReview ↗

Response to Reviewer Qf4R – Part 3 of 3

Q5: Importance of computational efficiency.

“Is reward calculation speed important, having less latency? Is the framework detrimental to be offline?”

Response: Low latency matters if, for example, attribution scores are needed to select advertisements shown alongside the answer, as proposed in Appendix B. For direct payments, MaxShapley can instead run after the answer is served, using the saved query, retrieved sources, and generated answer. Offline calculation does not limit that use; avoiding repeated generation still reduces the cost of processing many queries (Figure 4 in the revised manuscript).

Q6: Experimental claims.

“At section 1.1, LLM-as-a-judge utility function (Kendall-tau>0.5) and strong alignment with human annotations (Jaccard index>0.85) is written as achievement but they do not hold for HotpotQA and MS-Marco.”

Response: Thank you for pointing out that the dataset-specific results could be clearer. We first note that our more recent MaxShapley experiments using GPT-4.1-mini report substantially higher Jaccard and Kendall scores than the GPT-4.1-nano results in the submission (see table below). We have included these results in Appendix J of the revised manuscript.

Relevance-ranking performance using GPT-4.1-mini

MetricMethodHotPotQAMuSiQUEMS-MARCO
Jaccard@KMaxShapley0.922 ± 0.0190.977 ± 0.0130.759 ± 0.030
Jaccard@KFullShapley0.842 ± 0.0260.867 ± 0.0270.660 ± 0.031
Kendall τb\tau_bMaxShapley0.723 ± 0.0240.687 ± 0.0290.393 ± 0.028
Kendall τb\tau_bFullShapley1.000 ± 0.0001.000 ± 0.0001.000 ± 0.000

Each query is evaluated in three runs. Values are means ± standard errors.

We also have revised Section 1.1 to reflect the results more precisely. In particular, we have revised the statement to "LLM-as-a-judge utility function (Kendall-tau>0.5) and strong alignment with human annotations (Jaccard index>0.8) on HotPotQA and MuSiQUE". MS-MARCO is a challenging benchmark because its less curated source passages can be confusing, even for humans (Section 4.2): all methods have Jaccard <0.7, and all methods (except FullShapley’s self-comparison) have Kendall b<0.4.

Q7: Token accounting.

“Fig. 3 token consumption is not clear which consumption we are referring to. Also, error bar in the figure is not sure if it is calculated from 3 runs or 100 runs.”

Response: Token consumption denotes the total input and output tokens used per query, including answer generation and all LLM-based attribution stages. For MaxShapley, these include keypoint extraction, refinement, and support scoring; for FullShapley, they include generation and judging across source subsets. Each query is evaluated three times. The plotted values are means across queries, and the error bars represent standard errors across queries. We have clarified these definitions in the experimental setup.

Q8: Presentation improvements.

“Even though they are in the appendix, Fig. 4 and Fig. 5 helped me understand the main idea. I recommend this to be in the main explanation.”

“Minor point : MS-Marco and MS Marco is both used as a terminology.”

“Fig. 6 or Fig. 7 seems to be wrong (maybe Fig. 7). They look same.”

“‘See more details in Appendix 4.3.’ seems to be referring section 4.3 itself.”

Response: Thank you for these helpful presentation suggestions. We have brought the illustrations in Figures 4 and 5 into the main explanation of MaxShapley as Figure 3 in the revised manuscript, standardized the dataset name as MS-MARCO, replaced Figure 7’s duplicated content with the intended keypoint breakdown prompt, and corrected “Appendix 4.3” to “Appendix G.”

Reviewer_U1wrOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents MaxShapley, an algorithm for fair credit attribution in generative search, so that content providers can be compensated for their contributions to generated answers. The authors argue that Shapley-based attribution requires repetitive trials over subsets of sources, which makes it computationally infeasible in latency-sensitive search settings. In MaxShapley, an attribution LLM decomposes the answer into key points and assigns a support score for how well each source supports each key point. On about 100 queries from each of HotPotQA, MuSiQUE, and MS MARCO, with six sources per query, the paper compares attribution quality and token consumption with baselines such as FullShapley and KernelSHAP.

Critical strengths
  1. The paper investigates how to fairly pay content providers whose sources are used in generative search answers. It defines fairness with the four Shapley axioms.
  2. MaxShapley requires only black-box access and does not need token probabilities or log-likelihoods from the search LLM. Once the support scores are computed, the attribution step needs no subset sampling and no further LLM calls.
  3. On MuSiQUE with GPT-4.1 nano, MaxShapley reaches a Jaccard index of 0.88 with less token consumption compared to FullShapley.
  4. The authors manually re-annotated 100 HotPotQA, 100 MuSiQUE, and 95 MS MARCO queries with consensus labels from two annotators. They release these labels with the code, so other work on source attribution can reuse them.
Critical weaknesses
  1. MaxShapley checks whether a source supports a key point of the answer, not whether the source changed the answer, because it never reruns the search LLM without that source. So a source the LLM did not use can still get paid, and a source that supports a wrong claim in the answer gets credit.
  2. Because of the symmetry axiom, a copied page gets the same payment as its original, which is a known problem for Shapley payments [1]. This matters here because a site that copies an article would take part of the original author's payment.
  3. The experiments do not show what the Shapley step adds, because no simple rule on the same support scores is compared, such as adding up the scores of each source.
  4. Payments depend on how the LLM splits the answer into key points, since every key point gets the same weight. So a split into two key points carries twice the payment of a fact kept as one.

References

[1] Agarwal, Anish, Munther Dahleh, and Tuhin Sarkar. "A marketplace for data: An algorithmic solution." Proceedings of the 2019 ACM Conference on Economics and Computation. 2019.

Questions for authors
  1. How does MaxShapley compare with simple rules on the same support scores, such as adding up the scores of each source, or giving all the credit for a key point to its best source?
  2. How should MaxShapley pay a page that copies another retrieved page, and a source that supports a wrong claim in the answer?
5 comments
AuthorsOpenReview ↗

Response to Reviewer U1wr – Part 1 of 2

We thank you for your thoughtful review and hope that our responses to your concerns, together with new experiments on credit to unused sources, support a more favorable assessment of our work. We have included the additional results in the revised manuscript.

Q1: Credit for unused sources and sources supporting incorrect claims.

“MaxShapley checks whether a source supports a key point of the answer, not whether the source changed the answer, because it never reruns the search LLM without that source. So a source the LLM did not use can still get paid, and a source that supports a wrong claim in the answer gets credit.”

Response: Thank you for raising this concern. We can handle this by making use of MaxShapley’s modular design, which lets us customize its scorer without changing the allocation algorithm. We evaluate Relevance-MaxShapley, an instantiation that assigns zero support to sources identified as unused by the generator and halves scores for source–keypoint pairs judged unhelpful for answering the question. This yields 100% payment to used sources (see the table below), while preserving comparable ranking performance: mean Jaccard and Kendall scores change by at most 0.0325 and 0.0266, respectively. We have included this modification and evaluation in Section 4.4 and Appendix J.2 of the revised manuscript.

Regarding incorrect claims, similar concerns also arise in online advertising, where compensation generally depends on content use rather than claim-by-claim assessment of correctness (e.g. Google AdSense). The same principle applies to our setting, except that sources are aggregated into an answer by an AI model rather than accessed individually by a user. Furthermore, many questions lack a single objectively correct answer. Comprehensive factual verification and its evaluation extend beyond this work’s focus, and hence are left to future work.

Fraction of payment assigned to used sources (%, higher is better)

MethodHotPotQAMuSiQUEMS-MARCO
MaxShapley94.7 ± 1.097.0 ± 0.994.3 ± 0.8
Relevance-MaxShapley100.0 ± 0.0100.0 ± 0.0100.0 ± 0.0
FullShapley84.2 ± 1.775.5 ± 2.664.1 ± 1.9

Each query is evaluated in three runs. Values are means ± standard errors across query averages. The highest mean among the displayed methods for each dataset is shown in bold.

Q2: Copied sources and dilution of the original source’s payment.

“Because of the symmetry axiom, a copied page gets the same payment as its original, which is a known problem for Shapley payments [1]. This matters here because a site that copies an article would take part of the original author’s payment.”

“How should MaxShapley pay a page that copies another retrieved page, and a source that supports a wrong claim in the answer?”

Response: Thank you for bringing this up! We view this as a very interesting and important problem. As a starting point, we have added Appendix C.4, which discusses grouping sources that share the same provenance and treating each group as a single player, with support defined by the maximum among its members. Additional identical copies would then leave the group’s scores and hence allocation unchanged. Under a policy rewarding original providers, the group’s payment could be assigned to the identified original source. Compatible methods could support this preprocessing: Manku et al. (2007) detect near-duplicate web pages, while Evans et al. (2020) reconstruct the republication history of news content. Such methods could provide duplicate groups and provenance information before MaxShapley allocates credit, with its Shapley guarantees applying to the group-level utility. Evaluating this extension would require benchmarks with verified source provenance and controlled copying attacks, which is beyond the scope of this work. We therefore leave this extension and its evaluation to future work.

AuthorsOpenReview ↗

Response to Reviewer U1wr – Part 2 of 2

Q3: Benefits of Shapley step and comparison with simple rules.

“The experiments do not show what the Shapley step adds, because no simple rule on the same support scores is compared, such as adding up the scores of each source.”

“How does MaxShapley compare with simple rules on the same support scores, such as adding up the scores of each source, or giving all the credit for a key point to its best source?”

Response: Thank you for pointing this out. Simple rules can violate the Shapley fairness axioms. In particular, normalized score summation can violate additivity, while the best-source rule below can violate both symmetry and additivity. They therefore lack the full axiomatic fairness guarantees provided by MaxShapley.

To address your concern, we empirically evaluate two simple baselines: (i) Score summation, adding each source’s support scores across keypoints and normalizing the totals; and (ii) Best source, assigning each supported keypoint’s credit to its highest-scoring source, with ties split equally. We evaluate these baselines on HotPotQA, MuSiQUE, and MS-MARCO, with axiom violations aggregated across all three datasets. In short, all of the simple baselines violate (at least) additivity, which is important for consistent reward allocation whether attribution is computed per query or jointly across a batch–for example, when a platform calculates publisher compensation jointly for a day’s queries. MaxShapley achieves a comparable or better relevance ranking performance, while satisfying all four axioms. We have included this evaluation in Appendix J.3 of the revised manuscript.

Axiom violations

MethodEfficiencyNull playerSymmetryAdditivity
MaxShapley0 / 8880 / 2,9780 / 2920 / 835
Score summation0 / 8880 / 2,9780 / 292468 / 835
Best source0 / 8880 / 2,9783 / 292205 / 835

Entries report violations / eligible checks, aggregated across HotPotQA, MuSiQUE, and MS-MARCO.

Jaccard against benchmark source-relevance annotations

MethodHotPotQAMuSiQUEMS-MARCO
MaxShapley0.922 ± 0.0190.977 ± 0.0130.759 ± 0.030
Score summation0.913 ± 0.0210.973 ± 0.0140.761 ± 0.030
Best source0.919 ± 0.0190.977 ± 0.0130.766 ± 0.031

Kendall τb\tau_b against FullShapley with GPT-4.1-mini

MethodHotPotQAMuSiQUEMS-MARCO
MaxShapley0.723 ± 0.0240.687 ± 0.0290.393 ± 0.028
Score summation0.715 ± 0.0260.686 ± 0.0290.398 ± 0.028
Best source0.725 ± 0.0250.688 ± 0.0290.409 ± 0.029

Each query is evaluated in three runs. Values are means ± standard errors across query averages. The highest mean among the displayed methods for each dataset is shown in bold.

Q4: Impact of key point weights on payment. “Payments depend on how the LLM splits the answer into key points, since every key point gets the same weight. So a split into two key points carries twice the payment of a fact kept as one.”

Response: The total payment is fixed per query and distributed among sources in proportion to their Shapley values (Appendix B), so additional keypoints do not increase the overall payment. Additional keypoints therefore do not increase the total payment, although decomposition can affect how that budget is divided under uniform weighting. However, uniform weights are a simplifying choice in our experiments, rather than a requirement of MaxShapley (Equation 5). Keypoints can be assigned different weights to reflect their relative importance in the answer. In fact, our demo at fair-search.com already uses this weighted formulation.

Thanks to the author for the new baselines and the relevance maxshapely result.

Here, the new results show no gain from the Shapley step. On your new table, the score summation and best source differ from Maxshapley very insignificantly (at most 0.016) on every metric. Also the best source has the higher kendall score on all three datasets. So, I don't see any extra gain here.

For weakness 4, the response agree that the decomposition changes how the budget is split, but it reports no results with non-uniform weights.

Overall, I will keep my score.

Reviewer_UPXfOpenReview ↗
Rating
3
Confidence
1
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a tractable and fair credit assignment solution for generative searches. Unlike full Shapley values for credit assignment that is often computationally intractable due to utility function marginalizing over all possible permutations, the proposed MaxShapley utility operates over a certain number of key points extracted from the output whereby the support of the best supporting document for a key point is weighted by key point weightage and summed over all the key points. The authors present the O(mlogm)O(m log m) utility call as opposed to the full-shapley exponential time call. Furthermore, in empirical results, MaxShapley is shown to outperform Full Shapley in terms of jaccard index on three QA datasets for several times fewer attribution tokens by LLM judge. In terms of rank correlation though, the performance remains comparable for much less token cost.

Critical strengths
  1. The approach is quite computationally practical and quite applicable for extractive searches. The alignment with human annotation is also quite acceptable.
  2. MaxShapley seems mathematically rigorous and meets the basic fairness criteria.
  3. The analysis is quite thorough, covering many judge LLMs and discussing the limitations in terms of corroboration-aware attribution.
Critical weaknesses
  1. The analysis does not discuss the impact of potentially noisy and contradictory sources. This is especially important in real-time evolving scenarios.
  2. The authors also do not consider the impact of synergistic contributions, where two or more sources jointly support a key point, but they individually do not. MaxShapley does not seem to be effective in such scenarios. This could be quite relevant for reasoning-based searches.
Questions for authors
  1. How does MaxShapley function with synergistic and contradictory contributors?
1 comment
AuthorsOpenReview ↗

Response to Reviewer UPXf

We thank you for your positive evaluation of our work and would like to address your remaining concerns below.

Q1: Noisy and contradictory sources. “The analysis does not discuss the impact of potentially noisy and contradictory sources. This is especially important in real-time evolving scenarios.”

Response: Thank you for raising this concern. We can handle this by making use of MaxShapley’s modular design, which allows the scorer to be customized for different contribution criteria without changing the allocation algorithm. Specifically, we evaluate Relevance-MaxShapley: the scorer assigns zero support to sources identified as unused by the generator and halves scores for source–keypoint pairs judged unhelpful for answering the question. Our experiments show that this consistently increases the fraction of payment assigned to relevant sources across all three datasets (see the table below). We have included this modification and evaluation in Section 4.4 and Appendix J.2 of the revised manuscript.

Fraction of payment assigned to relevant sources (%, higher is better)

MethodHotPotQAMuSiQUEMS-MARCO
MaxShapley92.1 ± 1.796.5 ± 1.583.0 ± 2.4
Relevance-MaxShapley95.1 ± 1.497.5 ± 1.385.4 ± 2.4
FullShapley85.4 ± 1.778.6 ± 2.257.7 ± 2.2

Each query is evaluated in three runs. Values are means ± standard errors across query averages. The highest mean among the displayed methods for each dataset is shown in bold.

A contradiction-aware scorer could similarly identify or downweight conflicting evidence according to the platform’s chosen policy. However, evaluating such a scorer would require suitable contradiction data and annotations beyond those available in the current benchmarks, and hence is left to future work.

Q2: Synergistic contributions and joint reasoning.

“The authors also do not consider the impact of synergistic contributions, where two or more sources jointly support a key point, but they individually do not. MaxShapley does not seem to be effective in such scenarios.”

Response: As before, we can modify the utility and relevance prompt to handle these kinds of considerations. Although we would require a completely new data labeling to identify and measure performance on synergistic contributions, we can demonstrate how MaxShapley can be adapted in a closely related scenario: /corroborating sources/, where multiple sources support the same fact. In Appendix C.3, we already discuss an instantiation of MaxShapley that handles corroborating sources, where multiple sources supporting the same keypoint can increase confidence in that information.

We demonstrate the instantiation Corroboration-MaxShapley that incorporates the utility presented in Appendix C.3 with λ=0.5\lambda=0.5 and λ=1\lambda=1 and evaluates it on all three datasets. Corroboration-MaxShapley shifts more budget toward keypoints supported by multiple sources and consistently achieves higher mean Jain’s index across all three datasets, while retaining comparable Jaccard and Kendall scores. (Jain’s index is another measure of fairness in allocations.) We present HotPotQA below as a representative example and have included the complete results on all three datasets in Appendix C.3 of the revised manuscript.

HotPotQA: mean payment-budget share per keypoint (%)

Sources with positive support for a keypointMaxShapleyCorr.-MaxShapley (λ=0.5\lambda=0.5)Corr.-MaxShapley (λ=1\lambda=1)Queries
137.8 ± 1.134.6 ± 1.232.7 ± 1.394
234.8 ± 1.937.4 ± 2.039.7 ± 2.156
328.0 ± 2.135.4 ± 2.640.0 ± 3.023
427.4 ± 3.136.4 ± 4.241.1 ± 4.99
528.2 ± 2.639.5 ± 4.443.9 ± 5.66
629.2 ± 3.446.8 ± 6.454.1 ± 8.06

Each query is evaluated in three runs. Values are means ± standard errors across query averages. The highest mean among the displayed methods for each dataset is shown in bold.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.