acceptodds
Withdrawn from ICLR 2027

Efficient Decentralized Multi-task Dataset Valuation via Model Merging

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Accurate and efficient dataset valuation is essential for enabling fair and transparent data marketplaces, especially when multiple contributors provide data for training multi-task models. Most existing valuation methods, however, are limited to single-task settings, overlooking scenarios where a buyer aims to optimize performance across multiple downstream tasks. Moreover, traditional valuation approaches—such as Shapley-based or retraining-based methods—are computationally expensive and poorly suited for decentralized environments without a trusted central coordinator and with strict privacy constraints. We propose DMVM (Decentralized Multi-task Valuation via Model Merging), a novel framework that bypasses retraining and data sharing by leveraging task arithmetic to infer dataset contributions directly from model combinations. Instead of retraining or sharing raw data, DMVM quantifies how models trained on different datasets combine in parameter space to infer each dataset’s marginal utility across multiple tasks. This formulation yields a valuation process that is scalable, computationally efficient, and explicitly aligned with multi-task generalization behavior. To support decentralized deployment, we introduce a secure aggregation protocol that enables collaborative valuation without revealing individual model parameters or private data. We also provide theoretical error bounds characterizing the approximation quality of DMVM and validate our framework through comprehensive experiments on computer vision and natural language processing tasks.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Dataset valuation, Model Merging, Multi-task learning, Decentralized AI, Shapley value

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.40 over 5 reviews, above 9% of the others.
Reviewer_MtqeOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes DMVM, a decentralized framework for multi-task dataset valuation that replaces repeated coalition retraining with model merging. Models trained by individual sellers are merged to approximate coalition utilities, which are then used to estimate Shapley-style dataset values. The paper also introduces a privacy-aware aggregation protocol and derives bounds on the resulting approximation error. Experiments on vision and NLP benchmarks evaluate DMVM against retraining-based valuation methods in terms of rank correlation, value-estimation error, and runtime.

Critical strengths
  1. The paper presents a practical way to reduce the cost of Shapley-style dataset valuation by replacing repeated coalition retraining with model merging. The idea is simple and potentially useful, and the experiments show that the resulting estimates can remain reasonably close to exact Dataset Shapley values while requiring much less computation.
  2. The evaluation uses exact Dataset Shapley as the reference and reports both ranking metrics and numerical value-estimation error across vision and NLP settings. This gives a more complete picture of approximation quality than reporting ranking agreement alone.
Critical weaknesses
  1. The novelty is limited and largely compositional. Prior work has already studied Shapley-style dataset valuation, federated participant valuation, model merging as a proxy for training on dataset mixtures, and the theoretical connection between task arithmetic and multi-task optimization. DMVM mainly combines these ideas by using merged-model utilities within a Shapley valuation pipeline. I do not see a sufficiently strong conceptual or theoretical advance beyond this integration.
  2. Several central claims also hold only under narrower conditions than the paper's overall framing suggests. The valuation bound builds on an existing model-merging approximation bound and assumes full-batch gradient descent, whereas the experiments use stochastic optimization. The privacy protocol is restricted to a semi-honest setting and does not cover arbitrary collusion, and the paper does not provide direct privacy-attack evaluation or end-to-end measurements of decentralized communication cost. The privacy-aware estimator also loses substantial accuracy in small marketplaces. These limitations weaken the claims of theoretical fidelity and secure, scalable decentralized valuation.
Questions for authors
  1. Could the authors clarify the main conceptual novelty of DMVM beyond combining an existing model-merging proxy for dataset-mixture training with Shapley-style valuation? In particular, what technical problem does DMVM solve that is not already addressed by Merge to Mix or prior work on task arithmetic?
  2. The valuation guarantee relies on a model-merging bound derived under full-batch gradient descent, whereas the experiments use Adam or AdamW-based stochastic fine-tuning. Can the authors provide evidence or analysis showing that the bound remains informative in the optimization regime used in the experiments?
  3. The privacy protocol assumes semi-honest participants and explicitly excludes arbitrary collusion. What privacy guarantees, if any, hold when a buyer colludes with one or more sellers, or when the buyer attempts reconstruction from repeated queries over overlapping coalitions? Have the authors evaluated these attacks empirically?
  4. The scalability analysis beyond the directly evaluated marketplace sizes is largely based on projected costs from asymptotic formulas. Can the authors report end-to-end decentralized costs that include model transmission, mask exchange, coalition coordination, and the unmergeable transformations, rather than only merging and inference time?
1 comment

We thank the reviewer for recognizing the computational benefits of DMVM and the value of evaluating against exact Dataset Shapley. We address the four questions below.

Q1. Conceptual novelty beyond combining existing methods.

The central question is whether model merging preserves dataset marginal contributions, not merely overall mixture performance. Merge to Mix investigates whether merged models approximate training on dataset mixtures. This does not establish Shapley attribution fidelity. Let e(S)=u(S)−u^(S)e(S)=u(S)-\hat{u}(S) denote the utility-proxy error. The error in seller ii's marginal contribution is e(S∪{i})−e(S)e(S\cup\{i\})-e(S). Even a proxy that closely approximates each coalition's performance can misrank sellers if its errors vary across coalitions.

DMVM formulates, analyzes, and empirically tests this distinct property. Theorems 3.1–3.3 characterize how merging error, Monte Carlo sampling, and singleton exclusion affect valuation. Tables 2–3 compare both rankings and numerical values with retraining-based ground truth, and Appendix A.9 directly studies coalition-dependent errors. For nine NLP sellers, DMVM improves Kendall's τ\tau from 0.7030 to 0.7487 and reduces MSE from 0.002122 to 0.001668 relative to DU-Shapley (Table 2), while avoiding coalition retraining and access to sellers' raw data. This is not an automatic consequence of prior merging results or of the Shapley formula.

Q2. Theory under Adam/AdamW.

Theorem 3.1 is a conditional bound derived under full-batch gradient descent. We do not claim it formally extends unchanged to stochastic optimization, as already acknowledged in Appendix A.16. Its purpose is to characterize how utility-proxy error propagates into attribution error. Crucially, our empirical conclusions are evaluated independently of that assumption. Tables 2–3 measure ranking and value errors against exact Dataset Shapley using the actual optimizers and accuracy-based metrics. Appendix A.9 further examines the proxy errors responsible for that discrepancy. Thus, the paper provides both a theoretical characterization under stated conditions and direct evidence of practical attribution fidelity.

Q3. Privacy, collusion, and overlapping coalitions.

Our threat model is semi-honest without arbitrary collusion, as explicitly stated in Appendix A.16. We do not claim cryptographic protection against a colluding buyer and sellers. The proposed transformations are designed to protect against direct differencing of released coalition models, rather than provide an unconditional guarantee against reconstruction.

We also evaluated a concrete sorting-based alignment attack on eight NLP models fine-tuned on GLUE. Only 7% of neurons retained their pretrained rank on average, and no complete ordering was preserved. Sorting coalition weights before reconstructing an individual seller yielded a median error of ~90× the seller's task-vector norm, close to the ~95× error without undoing permutations. Thus, this reconstruction approach did not succeed in our evaluated models. These findings provide attack-specific empirical evidence, while stronger security under collusion remains beyond our stated threat model.

Q4. End-to-end decentralized cost.

Figure 2 already reports measured valuation runtimes, while Appendix A.13 explicitly presents projections for larger marketplaces. We additionally implemented Algorithm 1 using FLAN-T5-small (77M parameters), eight sellers, and 14 sampled coalitions of size at least two, profiling all steps on a CPU-only machine. Transformation coordination takes under 1 ms, transformation 0.09 s, and mask generation 0.6 s per seller per coalition. Buyer aggregation and mask verification take 0.3 s and 0.2 s per coalition. Total additional protocol computation is under one minute (~40 s).

Communication depends on the secure-aggregation implementation and network. Using standard shared-seed, pairwise-cancelling masks [1] would reduce full-size transfer to ~22 GB, or roughly 4 minutes at 1 Gbps including (de)serialization under the stated network assumptions. This is a bandwidth-based estimate for an optimized protocol, not a measured WAN runtime. Further communication optimization is deployment-dependent future work, orthogonal to DMVM's valuation methodology.

In summary, DMVM addresses a distinct attribution problem, verifies it against retraining-based ground truth, and explicitly characterizes the theoretical, privacy, and deployment trade-offs.

[1] Practical Secure Aggregation for Privacy-Preserving Machine Learning.

Reviewer_XC53OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes an aggregation protocol for estimating dataset Shapley values through model merging. The use case itself is a practical problem, since multi-site datasets are increasingly common, and aggregated analyses may provide greater benefits than analyses based on individual datasets.

However, the motivation for the specific methodology and experimental design is questionable. One of the biggest issues is that the authors do not sufficiently consider existing approaches to joint training and model aggregation, such as federated learning and meta-analysis, which raises several important questions about the necessity and advantages of the proposed framework.

I may consider changing my score if the authors could reasonably address the significant concerns with added experiments, but the overall status of this paper is not ready for acceptance without major revision.

Critical strengths

None.

Critical weaknesses

CW1. Insufficient consideration of existing joint-training approaches

Since SV is defined (at least in the context of this paper) based on the predictive performance of trained models, it is very unclear why the authors do not sufficiently consider existing approaches for joint model training/aggregation (e.g., federated learning, meta-analysis, etc.). These approaches have been well studied and should be considered as natural comparisons, including SVs computed using locally independent models and jointly trained models. The computational costs of these approaches also need to be compared. Given these issues, the numerical experiments fail to justify the significance or advantages of the proposed method.

Also given the existence of privacy-preserving training methods, the entire Section 3.2 also requires further justification regarding its necessity and comparisons with existing approaches.

CW2. Restrictive Feature-Space Assumptions

The assumption that all datasets DiD_i share the same feature space is way too strong, and the proposed aggregation relies heavily on this assumption. For example, if one site collects fewer features than another, the resulting models may have different input dimensions and parameter structures, making direct model merging considerably more challenging. This is a more severe challenge that remains difficult to resolve in the existing literature. However, the authors largely overlook this issue and rely on an overly idealized assumption.

Questions for authors

See CWs.

4 comments
AuthorsOpenReview ↗

Response to CW1

We thank the reviewer for the feedback. We respectfully disagree with the characterization that our experimental comparisons are insufficient and that our assumptions restrict DMVM to homogeneous datasets. We clarify both points below.

CW1. Comparison with federated learning and existing aggregation methods.

We would like to clarify the distinction between joint model training and dataset valuation. Federated learning (FL) addresses how to train a model collaboratively without centralizing raw data, whereas DMVM addresses how to estimate each seller's contribution before data acquisition, without repeated coalition training or a coordinated federated training process. These are fundamentally different objectives. Even with privacy-preserving FL, computing Shapley values through joint training would still require evaluating different seller coalitions, potentially involving repeated training or access to an existing federated training trajectory. DMVM avoids this requirement by reusing independently fine-tuned models.

Importantly, the requested comparisons are already included in our paper:

  • Tables 2–3: Comparisons with exact Dataset Shapley, MC-Dataset-Shapley, and DU-Shapley, where coalition utilities are obtained through joint training on seller datasets.
  • Appendix A.7, Table 4: Additional comparisons with GTG-Shapley, a federated participant-valuation method, as well as Leave-One-Out and Standalone valuation, which evaluates sellers independently.

The results support DMVM's effectiveness against both types of methods. At seven NLP sellers, DMVM achieves Kendall's τ=0.778\tau=0.778, compared with 0.724 for DU-Shapley (a joint-training-based valuation baseline) and 0.619 for GTG-Shapley (a federated valuation method). At nine sellers, DMVM achieves τ=0.749\tau=0.749 versus 0.703 and 0.389, respectively. DMVM also consistently outperforms these baselines, Leave-One-Out, and Standalone valuation in this evaluation across all tested seller counts.

Regarding computational efficiency, Table 1 explicitly compares the computational requirements, while Figure 2 reports the measured runtime–accuracy trade-off. The central advantage is that DMVM replaces repeated coalition training with model merging and inference, after a one-time local fine-tuning per seller.

Finally, privacy-preserving training does not eliminate the need for Section 3.2. Keeping raw data local during training and limiting the exposure of individual seller models during repeated coalition evaluations are distinct concerns. Our aggregation mechanism addresses the latter, with the proposed transformation designed to protect against direct differencing of coalition models.

AuthorsOpenReview ↗

Response to CW2

CW2. Restrictive feature-space assumptions.

We would like to clarify the distinction between compatible model architectures and homogeneous datasets or feature distributions. DMVM requires models to share a compatible parameterization for task-vector merging, as explicitly stated in Section 3.1. However, it does not require sellers to have identical datasets, distributions, or learning tasks.

Importantly, our framework primarily targets vision and language models, where shared pretrained architectures naturally provide compatible input representations. Language models process text through tokenized inputs, while vision models operate on standardized pixel representations. This allows models trained on substantially different domains and tasks to share the same architecture. The reviewer's example of datasets with different numbers of input features is more characteristic of heterogeneous tabular data, where feature schemas may differ across sources, rather than the vision and language settings studied here.

Our main experiments already demonstrate substantial data heterogeneity. The vision marketplace includes digit recognition, animal classification, scene classification, and satellite and aerial land-use imagery, while the NLP marketplace spans sentiment classification, paraphrase identification, and question answering. These datasets differ significantly in their domains, distributions, and prediction objectives, yet DMVM successfully estimates their contributions to the buyer's multi-task objectives.

We acknowledge that merging models with incompatible architectures or input representations requires additional techniques. However, this is a general limitation of conventional task-vector merging, not a restriction introduced by DMVM. Requiring a shared pretrained architecture is a standard and practically relevant setting in the model-merging literature and does not imply homogeneous seller datasets. This limitation is already explicitly acknowledged in Appendix A.16, where we also identify heterogeneous model merging as a potential direction for future work.

My evaluation remains the same. The rebuttal explanation diverges from the original questions raised and does not address the concerns (for instance, it fails to explain why using SV evaluation on a jointly trained model is not a comparable or better option).

We thank the reviewer for the follow-up. We would like to clarify that computing Shapley values using jointly trained models is precisely the ground-truth Dataset Shapley approach used in our paper, against which DMVM and all other valuation baselines are evaluated. By definition, this ground-truth reference has zero MSE, Kendall's τ=1\tau=1, and Spearman's ρ=1\rho=1 when compared against itself.

The purpose of DMVM is to approximate these ground-truth Shapley values without repeatedly training models on different seller coalitions, thereby substantially reducing computational costs while avoiding access to sellers' raw datasets.

We also encourage the reviewer to revisit Lines 347–350 of the manuscript, where Shapley valuation based on joint training is explicitly defined as the ground-truth reference.

Reviewer_YUr5OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DMVM, a model-merging-based approach to multi-task dataset valuation. The method aims to estimate the marginal utility of datasets across multiple tasks without directly sharing metadata, enabling efficient and scalable valuation of their contributions to multi-task generalization.

Critical strengths

Efficiently evaluating and utilizing private datasets is an important problem as access to additional high-quality training data becomes increasingly valuable. Estimating dataset utility without requiring direct metadata sharing addresses a practical need, and the multi-task setting adds relevance to this objective.

Critical weaknesses
  1. The introduction does not establish a coherent motivation. It begins with dataset valuation and existing approaches, shifts to model merging, and introduces the multi-task setting toward the end. The connections between these topics are insufficiently developed. Consequently, it is difficult to identify the specific limitation motivating DMVM, understand why model merging addresses it, or determine what additional challenges arise in the multi-task setting. This presentation also obscures the paper’s central contribution.
  2. The motivation for the model-merging component is unclear. The paper does not clearly explain which limitations of existing model-merging approaches are relevant to the proposed valuation framework. In particular, the efficiency bottleneck and the source of the claimed improvement need to be articulated more precisely. Is the contribution a new merging procedure, or the use of existing merging methods to reduce the cost of dataset valuation? The paper should distinguish these possibilities and explain which operation becomes less expensive and why.
  3. The framework combines established components, including Shapley-value estimation, model merging, and MPC-based secure aggregation. Combining existing techniques can constitute a meaningful contribution, but the paper needs to demonstrate what new insight or mechanism makes this combination nontrivial. As presented, the contribution beyond integrating these components is insufficiently clear. This leaves me unconvinced that the work offers sufficient technical novelty for ICLR.
Questions for authors

1.The introduction connects dataset valuation, multi-task learning, and model merging, but does not sufficiently articulate DMVM’s substantive advance over existing research. The paper already cites empirical and theoretical results supporting model merging as a proxy for joint training, while Section 3.1 primarily substitutes this proxy utility into the standard Shapley formula. 2.Theorem 3.1 relies on full-batch gradient descent and an approximation that neglects higher-order terms, whereas the experiments use Adam—a gap acknowledged in Appendix A.16. Moreover, the bound grows with the number of sellers and is derived for negative-loss utility, leaving unclear when seller rankings based on accuracy, as used in the experiments, are preserved. 3. The privacy-aware estimator excludes coalitions that would expose individual seller models, but Table 10 shows that this restriction substantially increases valuation error: with nine sellers, MSE rises from 0.0022 to 0.0066. Meanwhile, the scalability results in Appendix A.13 primarily extrapolate costs from training and inference times without fully accounting for model transmission and coordination overhead in secure aggregation. The current evidence therefore does not fully establish the trade-off between accuracy, privacy, and runtime in an actual decentralized deployment.

1 comment

We thank the reviewer for recognizing the importance of privacy-aware multi-task dataset valuation. We clarify our motivation, contribution, and evidence below.

1. Motivation, efficiency, and novelty.

DMVM asks: How can a buyer value datasets across multiple tasks without accessing sellers' raw data or retraining for each coalition? Our contribution is not a new merging operator, but a framework that reuses independently fine-tuned models and replaces coalition training with merging and inference. Table 1 specifies the computational savings, Figure 2 measures the runtime–accuracy trade-off, and MC-DMVM further reduces coalition evaluations.

We respectfully disagree that this merely inserts an existing proxy into Shapley estimation. Approximating joint-training performance does not imply preserving dataset contributions. Shapley values depend on differences between coalition utilities. Merging errors can distort these differences even when overall performance closely matches joint training. Prior merging studies do not establish attribution fidelity.

DMVM investigates this connection. Theorems 3.1–3.3 characterize merging, sampling, and coalition-exclusion errors. Tables 2–3 evaluate attribution against retraining-based ground truth, while Appendix A.9 examines coalition-dependent utility errors. For nine NLP sellers, DMVM improves Kendall's τ\tau from 0.7030 to 0.7487 and reduces MSE from 0.002122 to 0.001668 compared with DU-Shapley (Table 2), without accessing seller datasets. We agree that the introduction could emphasize this distinction more clearly, but formulating and validating merging-based multi-task attribution is itself a substantive contribution.

2. Theory versus practical evaluation.

Theorem 3.1 provides a conditional error bound under full-batch gradient descent and Lipschitz loss, not a universal guarantee for Adam or accuracy-based rankings. These limitations are already acknowledged in Appendix A.16. Importantly, Tables 2–3 directly measure ranking fidelity and numerical error against exact Dataset Shapley under the actual experimental settings. Thus, our empirical evidence does not depend on extending the theoretical assumptions to Adam.

3. Privacy–accuracy trade-off.

Singleton exclusion is optional for core DMVM, not an inherent limitation of merging-based valuation. The privacy-aware variant introduces it to avoid releasing singleton models. Theorem 3.3 bounds the additional error, and Appendix A.10, Table 10 evaluates it across marketplace sizes. Even after exclusion, at nine sellers, the estimator achieves Kendall's τ=0.790\tau=0.790 and Spearman's ρ=0.899\rho=0.899 against ground truth. Thus, privacy-sensitive terms can be excluded while retaining strong ranking agreement. This is a quantified design trade-off, not evidence that merging is an ineffective valuation proxy.

4. Runtime and decentralized communication.

Figure 2 already reports measured valuation runtimes, while Appendix A.13 projects larger marketplace sizes. We additionally implemented Algorithm 1 for FLAN-T5-small (77M parameters) with eight sellers and 14 sampled coalitions, profiling operations on a CPU-only machine. Transformation coordination takes under 1 ms, transformation 0.09 s, and mask generation 0.6 s per seller per coalition. Buyer aggregation and verification take 0.3 s and 0.2 s per coalition. Overall, protocol computation adds under one minute (~40 s).

Standard secure aggregation using pairwise-cancelling masks generated from shared seeds [1] would reduce full-size communication to approximately 22 GB, corresponding to ~4 minutes at 1 Gbps, including (de)serialization. This is a network-rate estimate, not a measured WAN runtime. Communication costs are deployment-dependent, and further optimizing them is future work, orthogonal to our core valuation contribution.

In summary, DMVM establishes and validates a distinct connection between model merging and dataset attribution, while characterizing its accuracy and deployment trade-offs.

[1] Practical Secure Aggregation for Privacy-Preserving Machine Learning.

Reviewer_fcs9OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes DMVM, a framework for estimating dataset-level Shapley values across a buyer's downstream tasks while sellers retain their training data. It approximates coalition utilities by merging independently fine-tuned models from a common checkpoint, with Monte Carlo sampling and a proposed protection protocol combining masked aggregation, parameter transformations, and exclusion of singleton-related marginal terms. The contributions include applying model merging to seller attribution and analyzing merging, sampling, and exclusion errors under a full-batch gradient-descent approximation and a Lipschitz loss assumption. Vision and language experiments show useful agreement with retraining-based seller valuations at reduced computational cost, while the coalition-exclusion ablation shows an accuracy cost for the protected estimator.

Critical strengths

The paper makes a credible case for cheaper dataset attribution without centralizing seller data. Merge to Mix: Mixing Datasets via Model Merging, Section 3.2, already uses merged models as mixture-training surrogates; DMVM tests whether this approximation also preserves marginal contributions, which mixture-performance correlation alone does not establish. Table 2 shows better NLP rankings and lower valuation error than DU-Shapley across the tested seller counts, while the vision rankings are less accurate than that baseline. Figure 2 and Appendix A.3 show the associated cost-error tradeoff and how sampling approaches the full proxy estimator. Together, these results support an effective approximation when coalition retraining is expensive. GTG-Shapley, Section 4.1, Eq. (6), already uses update-based reconstruction for valuation, but DMVM's independent seller fine-tunes avoid dependence on an iterative federated training trajectory. That combination and its attribution validation have value even without a new merging operator.

Critical weaknesses

The protocol does not establish the individual-model protection required by the paper's marketplace claim. This is distinct from the valid saving in retraining and raw-data access: Section 3.2 explicitly treats exposure of seller models as unacceptable and claims to prevent recovery across coalition releases. For the averaging case, its MLP mechanism releases PSASP_S A_S and BSPS⊤B_S P_S^\top, where AS,BSA_S,B_S are coalition averages. Consider a bias-free two-layer MLP with well-separated, ordered first coordinates in the public first-layer initialization and seller updates small enough to preserve that order. Sorting each released first-layer matrix reverses the permutation; applying the same ordering to second-layer columns aligns both coalition averages. Algorithm 1 releases every coalition of size at least two, so three sellers' pair-coalition outputs reveal seller 1 through W1(1)=A12+A13−A23W_1^{(1)}=A_{12}+A_{13}-A_{23}, with the corresponding formula for the second layer. A previously checked numerical construction recovers both matrices to floating-point precision. This is an existence counterexample for the MLP permutation branch, not an attack measurement on the paper's trained networks. It uses no singleton access, collusion, or protocol deviation and fits the semi-honest scope in Appendix A.16. Thus the evidence supports valuation with local raw data, but not the additional promise that sellers can withhold their individual models. This broken link matters because model protection is an explicit contribution and a condition of the intended deployment, rather than an optional extension imposed by the reviewer.

Questions for authors

Does the implemented protocol contain an additional transformation or coalition-release restriction absent from Section 3.2 and Algorithm 1 that rules out the reconstruction above? Please specify the assumption under which individual-model recovery is prevented.

1 comment

We thank the reviewer for this thoughtful and concrete analysis. The proposed attack relies on an important assumption: fine-tuning must preserve the neuron ordering inherited from the shared pretrained model, allowing coalition permutations to be reversed and individual models to be reconstructed from pairwise averages.

To investigate whether this assumption holds in practice, we evaluated eight NLP models fine-tuned on GLUE tasks. On average, only 7% of neurons retained their original rank, and the complete ordering was never preserved. More importantly, applying the reviewer's proposed reconstruction through A12+A13−A23A_{12}+A_{13}-A_{23} after sorting resulted in a median reconstruction error of approximately 90× the norm of the seller's task vector, nearly identical to direct differencing without reversing the permutations (95× error). These results demonstrate that the proposed attack does not successfully reconstruct individual seller models in our evaluated setting.

We clarify that our transformation is designed to protect against direct differencing of coalition models, rather than provide a formal cryptographic guarantee against arbitrary reconstruction attacks. Stronger protections, such as secure inference or trusted execution, remain promising directions for future work. Importantly, this discussion does not affect our central contribution of efficient, merging-based dataset valuation, which does not require access to sellers' raw data and whose accuracy is independently evaluated in the paper.

Reviewer_kzKBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes DMVM (Decentralized Multi-task Valuation via Model Merging), a framework for efficiently estimating dataset contributions in decentralized multi-task learning settings without sharing private data. The key idea is to use task arithmetic to merge independently fine-tuned models as a proxy for joint training, enabling Shapley-based dataset valuation without expensive retraining for different seller coalitions. The authors further introduce a privacy-aware secure aggregation protocol to protect individual models and provide theoretical bounds on valuation approximation errors. Experiments across computer vision and NLP benchmarks demonstrate that DMVM achieves strong agreement with ground-truth Shapley values in both seller rankings and numerical estimates while substantially reducing computational costs.

Critical strengths
  1. The paper presents a practical and efficient formulation. Replacing repeated coalition training with model merging substantially reduces the computational cost of Shapley-based valuation. This is particularly useful in decentralized settings where raw data cannot be shared.

  2. The paper has reasonably comprehensive evaluation. The experiments cover both vision and NLP tasks, multiple model architectures, and different seller configurations. The results generally show good agreement with ground-truth Shapley rankings.

Critical weaknesses
  1. The proposed method has insufficient privacy guarantees. While the proposed unmergeable transformations prevent straightforward parameter differencing across merged models, the privacy guarantees are not sufficiently established. In particular, it is unclear whether the transformations can resist more sophisticated reconstruction attacks, such as model alignment across different coalitions. The secure aggregation protocol also appears to rely on assumptions about participant behavior, without a clear analysis of potential collusion between the buyer and sellers. Since privacy preservation is a central motivation of the paper, a more rigorous security analysis or empirical attack evaluation is needed to substantiate these claims.

  2. The theoretical error bound relies on restrictive assumptions, including full-batch gradient descent and negligible higher-order terms, and may be too loose to provide meaningful guarantees in practical settings. More importantly, the privacy-aware Shapley formulation excludes coalitions of size less than two, which introduces systematic bias relative to the standard Shapley value. Although the authors provide an additional error bound for this approximation, its practical impact on valuation accuracy and seller rankings is not explicitly evaluated. This raises concerns about how reliably the proposed method approximates the intended ground-truth valuation, particularly for smaller marketplaces

Questions for authors
  1. Can the authors clarify the threat model and whether the privacy protocol remains secure against colluding sellers or model alignment attacks?

  2. How much does excluding coalitions of size less than two affect valuation accuracy? An ablation comparing the standard and privacy-aware Shapley estimates would be helpful.

  3. How sensitive is DMVM to the merging coefficient and the degree of task interference between seller models?

1 comment

We thank the reviewer for recognizing DMVM's computational efficiency, practical relevance, and comprehensive evaluation. We appreciate the constructive questions and address them below.

Q1. Privacy guarantees and reconstruction attacks.

Our protocol operates under a semi-honest, non-colluding threat model, as explicitly stated in Appendix A.16. Secure aggregation prevents the buyer from directly accessing individual seller models during aggregation, while the proposed transformations are designed to protect against direct differencing of overlapping coalition models, rather than provide a formal cryptographic guarantee against arbitrary reconstruction attacks.

To further investigate model-alignment attacks, we evaluated a sorting-based reconstruction strategy on eight NLP models fine-tuned on GLUE tasks. This attack relies on recovering neuron ordering from the shared pretrained model. However, only 7% of neurons retained their original rank on average, and the complete ordering was never preserved. Applying the sorting-based reconstruction yielded a median error of approximately 90× the seller's task-vector norm, nearly identical to direct differencing without reversing the permutations (95×). Thus, the attack did not successfully reconstruct individual models in our evaluated setting.

Arbitrary buyer–seller collusion is outside our stated threat model. Stronger protections, such as secure inference or trusted execution, remain valuable directions for future work.

Q2. Effect of excluding singleton coalitions.

We would like to clarify that the requested ablation is already provided in Appendix A.10, Table 10, where we directly compare DMVM-FULL and DMVM-EXCLUDE against exact Dataset Shapley across marketplace sizes from 3 to 9 sellers.

Importantly, at nine sellers, DMVM-EXCLUDE still achieves Kendall's τ=0.790\tau=0.790 and Spearman's ρ=0.899\rho =0.899, demonstrating strong agreement with ground-truth seller rankings despite excluding privacy-sensitive contributions. Its MSE decreases from 0.0483 at three sellers to 0.0066 at nine sellers, showing that the additional error becomes considerably smaller as the marketplace grows.

Theorem 3.3 also explicitly bounds the error introduced by this restriction. We emphasize that singleton exclusion is a design choice of the privacy-aware variant, not an inherent limitation of DMVM. Standard DMVM retains the full coalition information and achieves higher valuation accuracy. Thus, the paper already provides both theoretical and empirical characterizations of this privacy–accuracy trade-off.

Q3. Sensitivity to the merging coefficient and task interference.

We use α=1∣S∣\alpha=\frac{1}{|S|} in our experiments for simplicity and consistency, but DMVM is not restricted to this coefficient. Section 3.1 introduces a general scaling parameter, and Appendix A.1 provides the protocol for arbitrary α\alpha.

Optimizing the merging coefficient through grid search or other validation-based methods could improve the approximation of coalition utilities and, consequently, lead to more accurate Shapley estimates. This provides a natural opportunity for further improvement without modifying the valuation framework.

Regarding task interference, we agree that stronger interference can increase the utility approximation error and potentially the valuation MSE. However, accurate dataset valuation does not require perfectly reproducing every coalition's utility. Shapley values depend on marginal utility differences, so shared approximation errors can partly cancel across coalitions, preserving seller rankings even when individual utility estimates are imperfect. Appendix A.9 directly examines this behavior and finds that coalition-dependent error variation decreases substantially for larger coalitions.

More importantly, our experiments already demonstrate strong valuation accuracy across sellers from substantially different domains. In Table 2, DMVM achieves Kendall's τ=0.7795\tau =0.7795 for nine vision sellers and τ=0.7487\tau=0.7487 for nine NLP sellers, with MSE of 0.000747 and 0.001668, respectively. Appendix A.11 further demonstrates strong ranking agreement across heterogeneous multi-domain vision and language settings, including overlapping seller tasks.

Therefore, while stronger task interference can affect approximation quality, our results show that DMVM can preserve reliable seller rankings and low valuation error even across diverse tasks and domains. Optimizing the merging coefficient or incorporating more advanced merging methods provides additional opportunities to improve fidelity.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.