acceptodds
Under review as a conference paper at ICLR 2027

Shapley Data Valuation under Commitment: Adaptive Manipulation and Interaction Laundering

Abstract

Shapley-based data valuation can be manipulated when a provider adapts its submission to the coalition being evaluated. However, it remains unclear how restricting such adaptation affects manipulation and whether sufficiently strong commitment can eliminate overvaluation. We study this question using commitment partitions, which specify which coalition evaluations must reuse the same provider action. We characterize the loss in maximal manipulation gain caused by commitment and show that stronger commitment reduces, but does not eliminate, manipulation. In particular, even under one fixed submission, a multi-artifact provider can increase its Shapley payment while no coalition utility improves. We identify the underlying mechanism as interaction laundering, where interaction value shifts toward interactions in which the provider owns a larger share, and realize this behavior in both entity-matching and learned-representation settings. Finally, we show that adaptive and committed manipulation require different defenses. Reversible adaptive manipulation calls for removing exposed payment sensitivity, while one-sided committed degradation can be addressed by Best-Bundle, which applies free disposal before provider-level allocation.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Shapley value, data valuation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_CaPVOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary
  • The paper studies strategic manipulation of data valuation when providers own multiple artifacts and are paid the sum of their artifact's Shapley values.
Critical strengths
  • The paper shows that a provider can receive a higher Shapley payment from one fixed submission, even though none of the evaluated combinations becomes more useful and the performance with all submitted items remains unchanged. this is particularly interesting, because requiring providers to submit the same items throughout the evaluation does not by itself prevent overpayment.
  • experiments are convincing.
Critical weaknesses
  • The contribution over existing results is limited. for instance, the coefficient formula in Prop 2 can be found in Zheng et al (2025a, Sec 3). Moreover, applying Shapley to providers already prevents the payment increase covered by Prop 5 (merged results in table 6 also show this). Thus, the contribution is not particularly novel.
  • The practical relevance of the examples is a bit unclear. the provider introduces an incompatibility and receives separate credit for a way to undo it. in the representation expt, the other provider's prediction model is kept fixed, while a newly trained model has no reported accuracy loss from the changed representation (table 7). the example is valid under these choices, but it is unclear when an actual data valuation system would use this arrangement. in my view, this limits the paper's significance.
Questions for authors
  • see weaknesses
Reviewer_ejMAOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies data valuation. Therein, it positions the problem of data providers being able to change the utility of their datasets according to with what other data it is used together. If this is possible, then this offers the opportunity for "an attack" where actors can manipulate the aggregated attribution scores (Data Shapley values) for the bad actors benefit. The paper then argues that this can be solved with encouraging commitment inside the data valuation pipeline.

Critical strengths

None

Critical weaknesses

In my opinion, this paper is very hard to follow and understand its contribution. Overall, it contains many weaknesses:

  1. I have significant concerns about the motivation and practical relevance of this problem. In my view the problem is ill-posed and the main motivation of this paper hinges on (also see Q1, Q2 for side-questions). I do not really follow the motivation which reads as:

This creates a strategic problem when the evaluated data are supplied by providers who can choose how their artifacts are submitted or represented. Recent work has shown that a provider can increase its Shapley payment by adapting its report to the coalition being evaluated, even while lowering the utility of selected coalition evaluations (Zheng et al., 2025a). [lines 32-36]

First, the work Zheng et al. 2025a seems to be not peer reviewed, which is problematic as the main motivation of the work hinges on this. Second, for me, this setup seems very unrealistic and honestly is an ill-posed formulation of the data valuation problem. In data/dataset valuation, a data provider provides the dataset. That's it. The data is provided to the model builders or not. If it is provided, the model is fitted on the data in jointly with other datasets. This is done multiple times with multiple coalitions and all interactions and effects are then averaged into attribution scores, such as the Shapley value or (notably missing) the Banzhaf values to serve as proxies for the payment of the providers. Allowing the provider to choose how the data is provided is just not sensible. A report cannot. be changed. The provider does not report anything here, but just submits the data for the model builder to evaluate it. Particularly, conditioning a report on other subsets being part of a coalition is very confusing. All in all, I do not really see how this is a) a known issue or b) an issue that is practically relevant.

  1. The empirical evaluation is very small and only of synthetic nature. Typical data valuation settings involve multiple data providers even in dataset valuation settings. The here constructed games seem to be at most 6 players large, which in my opinion is very little compared to related works. Moreover, all constructed scenarios in the experiments are purely synthetic. This emphasizes my concern about the practical relevance and formulation of the problem. Even the "real data" experiments do not use any real multi-provider setting or any manipulation observed.

  2. The paper does not provide its source code for the submission. I wanted to check the empirical evaluation and the problem I had with the introduction (how it is implemented) and was hit with a not-working code link. The link (https://anonymous.4open.science/r/shapley-commitment-73D8/README.md) provided in the paper leads to a repo that is not accessible for reviewers. This makes this work not reproducible for me during review. Note, I tried to access the repository three times (once at the beginning of the review period, and twice towards the end).

  3. Finally, the list of references is quite small, but importantly most of the cited references are only preprint papers and not peer reviewed papers. A field such as data valuation, incentives for collaboration inside federated machine learning, and Shapley literature is vast and very prominent in top-tier venues such as ICLR. This makes it quite surprising that this is not the case for this submission.

Questions for authors

Q1: Already in the very first sentence of the introduction: I stumble over the word counterfactual coalitions. What do you mean by counterfactual? This is not very clear. I would understand this word a bit better in places of feature attribution where a coalitions are often instantiated with applying masking operations (marginal, baseline, etc.) and this creates "unrealistic" instances. This is not the case in data valuation. Each coalition is absolutely meaningful on its own: just the set of players providing data without any unrealistic cases. Q2: How could a data provider realistically change the output of the utility function which is evaluated by the central model builder collecting each provider's data? The coalitions' utility arises jointly from the mixture of the providers.

Reviewer_U5wvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies manipulation of Shapley-based data valuation under restrictions on coalition-dependent submissions. It introduces commitment partitions, characterizes the resulting manipulation gap, and constructs fixed-submission attacks through incompatibility and repair artifacts. It also proposes Best-Bundle, combining sub-bundle selection with provider-level allocation, and evaluates attacks and defenses in entity-matching, representation, and controlled settings.

Critical strengths

The fixed-submission construction usefully shows that commitment alone does not eliminate manipulation of Atomic Shapley. The experiments include held-out evaluation, query-matched search controls, and explicit checks of the defense assumptions. The authors clearly acknowledge the classical components of Best-Bundle and the effectiveness of existing ownership-aware defenses.

Critical weaknesses

The commitment analysis and fixed-submission constructions provide useful extensions of Zheng et al., and Best-Bundle adds the ability to discard harmful artifacts. Nevertheless, the underlying negative-coefficient vulnerability and provider-level defense are already established. While the experiments substantiate the proposed attack interfaces, I find the additional conceptual and practical insight insufficiently substantial for acceptance. My concern is the significance of the incremental contribution, rather than the correctness of the constructions or an omission of existing defenses.

Questions for authors

No specific questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.