In my opinion, this paper is very hard to follow and understand its contribution. Overall, it contains many weaknesses:
- I have significant concerns about the motivation and practical relevance of this problem. In my view the problem is ill-posed and the main motivation of this paper hinges on (also see Q1, Q2 for side-questions). I do not really follow the motivation which reads as:
This creates a strategic problem when the evaluated data are supplied by providers who can choose how their artifacts are submitted or represented. Recent work has shown that a provider can increase its Shapley payment by adapting its report to the coalition being evaluated, even while lowering the utility of selected coalition evaluations (Zheng et al., 2025a). [lines 32-36]
First, the work Zheng et al. 2025a seems to be not peer reviewed, which is problematic as the main motivation of the work hinges on this. Second, for me, this setup seems very unrealistic and honestly is an ill-posed formulation of the data valuation problem. In data/dataset valuation, a data provider provides the dataset. That's it. The data is provided to the model builders or not. If it is provided, the model is fitted on the data in jointly with other datasets. This is done multiple times with multiple coalitions and all interactions and effects are then averaged into attribution scores, such as the Shapley value or (notably missing) the Banzhaf values to serve as proxies for the payment of the providers. Allowing the provider to choose how the data is provided is just not sensible. A report cannot. be changed. The provider does not report anything here, but just submits the data for the model builder to evaluate it. Particularly, conditioning a report on other subsets being part of a coalition is very confusing. All in all, I do not really see how this is a) a known issue or b) an issue that is practically relevant.
-
The empirical evaluation is very small and only of synthetic nature. Typical data valuation settings involve multiple data providers even in dataset valuation settings. The here constructed games seem to be at most 6 players large, which in my opinion is very little compared to related works. Moreover, all constructed scenarios in the experiments are purely synthetic. This emphasizes my concern about the practical relevance and formulation of the problem. Even the "real data" experiments do not use any real multi-provider setting or any manipulation observed.
-
The paper does not provide its source code for the submission. I wanted to check the empirical evaluation and the problem I had with the introduction (how it is implemented) and was hit with a not-working code link. The link (https://anonymous.4open.science/r/shapley-commitment-73D8/README.md) provided in the paper leads to a repo that is not accessible for reviewers. This makes this work not reproducible for me during review. Note, I tried to access the repository three times (once at the beginning of the review period, and twice towards the end).
-
Finally, the list of references is quite small, but importantly most of the cited references are only preprint papers and not peer reviewed papers. A field such as data valuation, incentives for collaboration inside federated machine learning, and Shapley literature is vast and very prominent in top-tier venues such as ICLR. This makes it quite surprising that this is not the case for this submission.