- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes Activation Sharing Federated Learning (ASFL), a federated optimization approach intended to reduce client drift under heterogeneous client data distributions. ASFL exchanges intermediate activations and associated labels between participating clients. Each client subsequently performs partial-model updates on both local and received activations before conventional full-model training and aggregation.
The authors introduce a layer-selection heuristic balancing computational/communication cost against gradient variance, analyze convergence for smooth non-convex objectives, and investigate optional differential privacy mechanisms. Experiments on CIFAR-10, CIFAR-100, SVHN, and FEMNIST report improved final accuracy over several federated learning baselines under fixed communication rounds.
Critical strengths
- Meaningful empirical improvements with a useful causal ablation.
ASFL achieves consistent accuracy improvements across the four principal benchmarks. The CIFAR-10 results are particularly encouraging. Importantly, Appendix Table 14 compares ASFL against a partial-training variant without remote activations and shows a 9.04 percentage-point improvement from incorporating the shared representations. This helps isolate the benefit of the proposed information-sharing mechanism.
- A practical approach to reducing the cost of representation sharing.
Training only the model tail using received activations, combined with reuse of a small activation pool, offers a potentially useful mechanism for reducing redundant computation. The ablations on sharing layers and activation quantities help characterize this trade-off.
Critical weaknesses
- The convergence analysis contains a potentially invalid bound.
In the proof of Lemma 4 (Appendix A.2, page 16), the discrepancy between client gradients and the global gradient is bounded using the within-client stochastic variance (\sigma_z^2). This discrepancy is instead associated with between-client gradient heterogeneity, represented by (\sigma_G^2) in the assumptions.
The bound does not generally follow from the stated assumptions. Moreover, the analysis should justify its gradient unbiasedness and independence assumptions under the finite, repeatedly reused activation pools implemented in ASFL.
Since this lemma is used in the subsequent theoretical derivations, the convergence and linear-speedup claims require reconsideration.
- The efficiency and privacy claims are not yet sufficiently established.
The reported timing results measure local computation rather than complete federated training time, excluding the additional peer-to-peer communication and synchronization costs. Communication overhead is also substantial on certain benchmarks; for example, it nearly doubles relative to FedAvg on SVHN.
The differential privacy analysis primarily considers perturbations to transmitted activations, while the protocol also shares corresponding labels and continues to exchange model updates. The scope of the claimed privacy guarantee, the treatment of labels, and privacy composition across rounds require clarification. The limited model-inversion experiment does not establish broader privacy protection.
The comparison to earlier feature-sharing approaches, particularly FedFed and FedMix, should also be strengthened to clarify the specific novelty of ASFL.
Questions for authors
Q1 — Theoretical correctness: Can the authors clarify or correct the bound in Lemma 4 and show how the convergence argument applies to the actual activation-reuse procedure? Does the claimed linear-speedup result continue to hold under partial participation with a fixed number of active clients?
Q2 — System efficiency: Can the authors provide end-to-end convergence measurements against wall-clock time and total bytes communicated, including peer transfers and synchronization? How does ASFL compare against appropriate baselines under matched communication budgets?
Q3 — Privacy: What is the precise adjacency relation and adversarial threat model for the stated DP guarantee? Are the transmitted labels considered public? How are repeated releases and model-update disclosures accounted for? Can the authors substantiate their privacy claims beyond the limited reconstruction examples?
We appreciate the constructive review. We hope our responses address your concerns, and we would be grateful if you could reconsider your evaluation.
Q1, W1: Can the authors clarify or correct the bound in Lemma 4 and show how the convergence argument applies to the actual activation-reuse procedure? Does the claimed linear-speedup result continue to hold under partial participation with a fixed number of active clients?
Q2, W2: System efficiency: Can the authors provide end-to-end convergence measurements against wall-clock time and total bytes communicated, including peer transfers and synchronization?
Q3, W2: Privacy: 1) What is the precise adjacency relation and adversarial threat model for the stated DP guarantee? 2) Are the transmitted labels considered public? 3) How are repeated releases and model-update disclosures accounted for? 4) Can the authors substantiate their privacy claims beyond the limited reconstruction examples?
Table 1. Accuracy with PixelDP and label DP (Laplacian ϵ=2.0, Gaussian (ϵ,δ)=(0.5,0.05), “–”: no label DP)
Table 2. Inversion attacks (Laplacian ϵ=0.2, Gaussian (ϵ,δ)=(0.5,0.01))