- Rating
- 2(−1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates whether Shapley values can be accurately recovered in privacy-preserving federated learning when only two measurements are available for each client: its standalone gain and removal loss. The results show that Shapley values can be exactly recovered when interactions are limited to pairs of clients. However, when higher-order interactions are present, Shapley values may no longer be uniquely identified.
Critical strengths
S1. The research question is underexplored and meaningful for improving the efficiency and security of contribution evaluation in FL.
S2. The research question is answered with a complete set of theoretical characterizations.
S3. The presentation is good, and the claims are carefully stated.
Critical weaknesses
W1. The key contribution Theorem 6 gives exact intervals only when the true game has nonnegative effects through order three. However, only 2.2% of the empirical observations satisfy this assumption.
W2. The scope of the experiments is quite limited. Only three datasets and single-round FedAvg are considered, which limits the generalizability of the empirical findings.
Questions for authors
Q1. How can we calculate the L value in Theorem 7 for neural networks?
4 comments
Q1. How can we calculate the value in Theorem 6 for neural networks?
The MNIST head-update experiment uses the explicit global bound in Frobenius norm.
For the frozen encoder, only the head matrix varies, and the logits are . Unit-bounded augmented embeddings give and in Frobenius parameter norm and Euclidean logit norm. Appendix C.1 derives the global negative-cross-entropy bound , so substitution permits . Section 5.4 and Appendix C.3 apply that analytic constant to the head-update contribution game; the encoder parameters remain fixed.
For a fully updated smooth network, the regional derivative calculation supplies a sufficient bound.
First enclose the coalition models using the public checkpoint and weighted update-norm caps . At maximum common scale , the triangle inequality gives a ball around the checkpoint with radius . The utility must be three times continuously differentiable near that region. This construction uses public caps and requires no individual update directions or extra coalition evaluations.
Next use the public input bound, regional weight bounds and analytic activation derivatives in the chain and product rules through the layers. Let bound the network logit map's first three parameter derivatives, and let bound the utility's first three logit derivatives. The bounds must hold throughout the region and for every allowed evaluation input and label. The third-order chain rule gives
.
A uniform example-level bound also bounds the empirical mean utility; alternatively, average separate valid example-level bounds. Pointwise automatic differentiation at the checkpoint or sampled models does not establish the regional supremum. Appendix B.4 gives this construction and discusses the conservatism of the tested full-network bounds. ReLU activation boundaries require a separate smoothness argument.
Reminder
Dear Reviewer,
We have posted detailed responses to your comments and uploaded a revised manuscript. We clarified the distinction between the structural identified set and the smooth certificates used in the experiments, added a clipping and scale sensitivity analysis and an MNIST head-update study, and explained how to obtain the required derivative bound for neural networks (Appendices B.4 and C.2–C.3).
As the discussion period ends on November 18, we would be grateful to hear whether these revisions address your concerns, or whether any points would benefit from further clarification.
Thank you for your time and consideration.
We thank the authors for their detailed response.
W1: My concern has been addressed.
W2: I still do not see any multi-round results.
Q1: For a fully updated smooth network, the computed value of L appears to be too large, so large that the certificate becomes completely uninformative. I suspect this is also why the MNIST experiment updates only the linear logistic head. This raises my concerns about the practical benefit of the paper’s contributions.
Thank you for taking the time to read and review our work. We have uploaded an updated version of the paper. The theorem numbering and all section, table and appendix references below refer to that version. Original Theorems 6–8 are now Theorems 5–7.
The replies below distinguish the structural identified set from the operational smooth certificate, report the added comparisons, and give the neural derivative calculation.
W1. The structural theorem gives exact intervals only when the true game has nonnegative effects through order three, while the reported 2.2% figure suggests limited empirical applicability.
The structural theorem characterizes information left unresolved by exact recovery. In Figure 1, two positive third-order games share si=1, mi=7 and V=14 but have different top contributors. Theorem 5 identifies every compatible Shapley vector in the class; for that example each coordinate's sharp interval is [10/3,4]. It determines the remaining uncertainty, including the rankings common to all compatible vectors.
The 2.2% figure measures endpoint compatibility with this class, not true-game membership. All six compatible observations in the diagnostic have true Shapley vectors outside at least one structural interval. Section 3.3 therefore presents the result as a structural benchmark requiring independent membership justification, and Appendix C.6 states what the compatibility test establishes.
The experiments use the separate smooth certificate, which permits signed interactions of arbitrary order under public curvature and update controls. The structural compatibility rate does not assess that certificate's validity. Sections 5.3–5.4 instead measure its informative decisions and retained progress, as reported under W2.
W2. The original experiments cover only three datasets and single-round FedAvg, limiting their empirical generality.
We addressed operating-range generality with a paired sensitivity on both logistic datasets and added a learned-representation task on MNIST. The sensitivity reuses each profile across clip/scale choices, so the comparison measures the controls' effect with the underlying updates fixed. At full scale, the highlighted settings retain 87.6% and 86.3% median unclipped round gain and certify 482/560 and 471/560 pairs, with 14/20 and 13/20 winners. Section 5.3, Table 1, Figure 2 and Appendix C.2 give the full comparison and separate coordinate-only from combined certificates.
The added MNIST task evaluates a validation-only selection rule and certification for head updates over a fixed learned encoder. The original coordinate rule certifies 526/560 test pairs, with at least 23/28 in every replicate and median 27/28. Applying the combined rule at the same selected settings gives 533/560 pairs and 18/20 winners. Section 5.4 and Appendix C.3 report the original study, and Appendix C.7 reports the fixed-selection refinement. The partition replicates share image pools; these counts describe the evaluated completed rounds.
The formal results hold over their stated game classes; the experiments test whether their certificates give useful decisions in the evaluated rounds. The eight-client design permits exhaustive coalition evaluation to audit the intervals against exact Shapley, while the scorer still receives only endpoints and public controls. The sensitivity and learned-representation study establish the measured operating range and its progress cost. Section 5 and Appendix C now make that theory–experiment relationship explicit.