- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes DMVM, a decentralized framework for multi-task dataset valuation that replaces repeated coalition retraining with model merging. Models trained by individual sellers are merged to approximate coalition utilities, which are then used to estimate Shapley-style dataset values. The paper also introduces a privacy-aware aggregation protocol and derives bounds on the resulting approximation error. Experiments on vision and NLP benchmarks evaluate DMVM against retraining-based valuation methods in terms of rank correlation, value-estimation error, and runtime.
Critical strengths
- The paper presents a practical way to reduce the cost of Shapley-style dataset valuation by replacing repeated coalition retraining with model merging. The idea is simple and potentially useful, and the experiments show that the resulting estimates can remain reasonably close to exact Dataset Shapley values while requiring much less computation.
- The evaluation uses exact Dataset Shapley as the reference and reports both ranking metrics and numerical value-estimation error across vision and NLP settings. This gives a more complete picture of approximation quality than reporting ranking agreement alone.
Critical weaknesses
- The novelty is limited and largely compositional. Prior work has already studied Shapley-style dataset valuation, federated participant valuation, model merging as a proxy for training on dataset mixtures, and the theoretical connection between task arithmetic and multi-task optimization. DMVM mainly combines these ideas by using merged-model utilities within a Shapley valuation pipeline. I do not see a sufficiently strong conceptual or theoretical advance beyond this integration.
- Several central claims also hold only under narrower conditions than the paper's overall framing suggests. The valuation bound builds on an existing model-merging approximation bound and assumes full-batch gradient descent, whereas the experiments use stochastic optimization. The privacy protocol is restricted to a semi-honest setting and does not cover arbitrary collusion, and the paper does not provide direct privacy-attack evaluation or end-to-end measurements of decentralized communication cost. The privacy-aware estimator also loses substantial accuracy in small marketplaces. These limitations weaken the claims of theoretical fidelity and secure, scalable decentralized valuation.
Questions for authors
- Could the authors clarify the main conceptual novelty of DMVM beyond combining an existing model-merging proxy for dataset-mixture training with Shapley-style valuation? In particular, what technical problem does DMVM solve that is not already addressed by Merge to Mix or prior work on task arithmetic?
- The valuation guarantee relies on a model-merging bound derived under full-batch gradient descent, whereas the experiments use Adam or AdamW-based stochastic fine-tuning. Can the authors provide evidence or analysis showing that the bound remains informative in the optimization regime used in the experiments?
- The privacy protocol assumes semi-honest participants and explicitly excludes arbitrary collusion. What privacy guarantees, if any, hold when a buyer colludes with one or more sellers, or when the buyer attempts reconstruction from repeated queries over overlapping coalitions? Have the authors evaluated these attacks empirically?
- The scalability analysis beyond the directly evaluated marketplace sizes is largely based on projected costs from asymptotic formulas. Can the authors report end-to-end decentralized costs that include model transmission, mask exchange, coalition coordination, and the unmergeable transformations, rather than only merging and inference time?
Response
We thank the reviewer for recognizing the computational benefits of DMVM and the value of evaluating against exact Dataset Shapley. We address the four questions below.
Q1. Conceptual novelty beyond combining existing methods.
The central question is whether model merging preserves dataset marginal contributions, not merely overall mixture performance. Merge to Mix investigates whether merged models approximate training on dataset mixtures. This does not establish Shapley attribution fidelity. Let e(S)=u(S)−u^(S) denote the utility-proxy error. The error in seller i's marginal contribution is e(S∪{i})−e(S). Even a proxy that closely approximates each coalition's performance can misrank sellers if its errors vary across coalitions.
DMVM formulates, analyzes, and empirically tests this distinct property. Theorems 3.1–3.3 characterize how merging error, Monte Carlo sampling, and singleton exclusion affect valuation. Tables 2–3 compare both rankings and numerical values with retraining-based ground truth, and Appendix A.9 directly studies coalition-dependent errors. For nine NLP sellers, DMVM improves Kendall's τ from 0.7030 to 0.7487 and reduces MSE from 0.002122 to 0.001668 relative to DU-Shapley (Table 2), while avoiding coalition retraining and access to sellers' raw data. This is not an automatic consequence of prior merging results or of the Shapley formula.
Q2. Theory under Adam/AdamW.
Theorem 3.1 is a conditional bound derived under full-batch gradient descent. We do not claim it formally extends unchanged to stochastic optimization, as already acknowledged in Appendix A.16. Its purpose is to characterize how utility-proxy error propagates into attribution error. Crucially, our empirical conclusions are evaluated independently of that assumption. Tables 2–3 measure ranking and value errors against exact Dataset Shapley using the actual optimizers and accuracy-based metrics. Appendix A.9 further examines the proxy errors responsible for that discrepancy. Thus, the paper provides both a theoretical characterization under stated conditions and direct evidence of practical attribution fidelity.
Q3. Privacy, collusion, and overlapping coalitions.
Our threat model is semi-honest without arbitrary collusion, as explicitly stated in Appendix A.16. We do not claim cryptographic protection against a colluding buyer and sellers. The proposed transformations are designed to protect against direct differencing of released coalition models, rather than provide an unconditional guarantee against reconstruction.
We also evaluated a concrete sorting-based alignment attack on eight NLP models fine-tuned on GLUE. Only 7% of neurons retained their pretrained rank on average, and no complete ordering was preserved. Sorting coalition weights before reconstructing an individual seller yielded a median error of ~90× the seller's task-vector norm, close to the ~95× error without undoing permutations. Thus, this reconstruction approach did not succeed in our evaluated models. These findings provide attack-specific empirical evidence, while stronger security under collusion remains beyond our stated threat model.
Q4. End-to-end decentralized cost.
Figure 2 already reports measured valuation runtimes, while Appendix A.13 explicitly presents projections for larger marketplaces. We additionally implemented Algorithm 1 using FLAN-T5-small (77M parameters), eight sellers, and 14 sampled coalitions of size at least two, profiling all steps on a CPU-only machine. Transformation coordination takes under 1 ms, transformation 0.09 s, and mask generation 0.6 s per seller per coalition. Buyer aggregation and mask verification take 0.3 s and 0.2 s per coalition. Total additional protocol computation is under one minute (~40 s).
Communication depends on the secure-aggregation implementation and network. Using standard shared-seed, pairwise-cancelling masks [1] would reduce full-size transfer to ~22 GB, or roughly 4 minutes at 1 Gbps including (de)serialization under the stated network assumptions. This is a bandwidth-based estimate for an optimized protocol, not a measured WAN runtime. Further communication optimization is deployment-dependent future work, orthogonal to DMVM's valuation methodology.
In summary, DMVM addresses a distinct attribution problem, verifies it against retraining-based ground truth, and explicitly characterizes the theoretical, privacy, and deployment trade-offs.
[1] Practical Secure Aggregation for Privacy-Preserving Machine Learning.