acceptodds
Withdrawn from ICLR 2027

Efficient Private Federated Learning with Error Feedback under Random Allocation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Gradient sparsification is widely used in differentially private federated learning to reduce communication cost. However, sparsification drops coordinates that may carry useful gradient information. Error feedback can recover this information by storing the dropped coordinates and reusing them later, but its effectiveness is limited by how often a client participates: under partial client participation, each client is selected infrequently, so the dropped coordinates become stale and rarely reintroduced. Under differential privacy two further problems arise: adding the stored coordinates back inflates the update norm before clipping, and storing them after the noise carries that noise into later rounds. In this paper, we propose Clean Error Residual Feedback (CERF), where each client stores the coordinates dropped by sparsification from its clipped model update before noise is added, overwrites them at each participation, and feeds them back only at the coordinates the current mask keeps. To tackle the problem of infrequent participation, we use random allocation for client selection, which lets each client participate more regularly while still benefiting from the privacy amplification of subsampling. We derive the privacy and convergence analysis of CERF, showing the trade-off between compression, staleness, and noise terms. Empirical experiments show that at a fixed privacy level, there exists a sparsification rate and allocation cycle length where CERF matches or improves the accuracy of DP-FedAvg at a fraction of the communication and energy cost per client.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Federated Learning, Gradient Sparsification, Differential Privacy, Error Feedback, Random Allocation, Optimization

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_NtavOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper identifies three major issues when applying traditional error feedback to differentially private federated learning and proposes Clean Error Residual Feedback which stores the coordinates dropped by sparsification and overwrites and feeds them back only at the coordinates the current mask keeps. Random allocation ensures that each client participates once per cycle. The paper presents privacy and convergence analyses and experiments on Fashion-MNIST and FEMNIST.

Critical strengths
  1. The empirical study is extensive, which includes both IID Fashion-MNIST and heterogeneous FEMNIST and evaluate several compression rates and participation schedules. The without-feedback comparison is an informative ablation study that shows CERF contributes beyond sparsification alone.
Critical weaknesses
  1. The central theoretical result has unvalidated steps. Theorem 3 is used as a narrative hand-wave rather than a mathematical step. The authors state that Random Allocation is bounded by Poisson subsampling via Theorem 3, but then they proceed to prove Theorem 1 using standard Poisson subsampling with q=1/Kq = 1/K, completely ignoring the rate inflation η\eta and additive parameters (Kδ0+δK\delta_0 + \delta) required by Theorem 3.

  2. Lemma 4 holds with strict constraints on the Rényi order α\alpha. However, in Appendix C.2 line 718, the authors set α=1+2log⁡(1/δ)ϵ\alpha = 1 + \frac{2\log(1/\delta)}{\epsilon} without proving that this specific α\alpha falls within the allowed validity domain of Lemma 4 for all parameter choices.

  3. In Remark 2, The Jensen step bounding the expected masked residual norm is valid, but its substitution into Eq. (11) is not justified. The function h(x)=Cx/[∥g∥(∥g∥−x)]h(x)=Cx/[\|g\|(\|g\|-x)] is increasing and strictly convex, so an upper bound on E[X]\mathbb E[X] does not provide the claimed upper bound on E[h(X)]\mathbb E[h(X)].

Questions for authors

Please refer to weakness.

Reviewer_8V9ROpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Clean Error Residual Feedback (CERF), a method combining gradient sparsification, differential privacy, and error feedback in federated learning. CERF stores residuals before adding DP noise, overwrites them at each participation, and selectively reuses them in subsequent updates. The authors also adopt random allocation to control residual staleness under partial client participation. Privacy and convergence analyses are provided, along with experiments on Fashion-MNIST and FEMNIST to evaluate accuracy, communication, and energy efficiency.

Critical strengths
  1. The paper considers the interaction between gradient sparsification, error feedback, and differential privacy in federated learning. In particular, the discussion of residual staleness and clipping effects under partial client participation addresses a relevant practical problem.

  2. The paper investigates different sparsification rates and client participation cycles under fixed privacy budgets. The experimental ablations provide some useful observations about how residual feedback affects model accuracy and communication efficiency.

Critical weaknesses
  1. The novelty of the proposed method appears to be limited, as CERF mainly combines existing techniques, including gradient sparsification, differential privacy, and error feedback. Its main design choices, such as storing clean residuals, overwriting rather than accumulating them, and selectively feeding them back, are relatively incremental modifications to existing error-feedback methods. More importantly, the paper does not sufficiently justify why an additional error-feedback mechanism is necessary to compensate for the information loss caused by sparsification. It remains unclear whether this problem could be addressed more directly by improving the sparsification strategy itself or adopting alternative compression methods.

  2. The client participation assumption is rather restrictive and may not reflect practical federated learning systems. CERF adopts random allocation, requiring every client to participate exactly once within each cycle of K rounds. However, clients in real-world federated learning systems may be unavailable, unwilling to participate, or disconnected. Random allocation guarantees scheduled selection but not actual participation. This assumption also weakens the paper's motivation of addressing uncertain client participation, since the method effectively replaces unpredictable participation with a predefined schedule.

  3. There are concerns about the validity and practical relevance of the theoretical analysis. First, Theorem 2 assumes that clipping is inactive, although reducing clipping-induced signal loss is one of the main motivations of CERF. Therefore, the convergence analysis does not directly characterize the benefits of the proposed mechanism when clipping actually occurs. Second, the proof of Lemma 2 appears to contain an error. In Equation (30) of Appendix E.2, the clipped update is treated as independent of the random sparsification mask. However, the clipped update depends on the current mask because the residual is added before clipping. Consequently, the bound in Equation (14) does not hold in general. For example, with d=2, k=1, C=1, g=(1,1), and a previous residual e=(-1,0), the expected squared residual is 0.75, exceeding the claimed upper bound of 0.5. Since Lemma 2 is used in the convergence proof, the validity of the subsequent theoretical results requires further justification.

  4. The experimental evaluation is insufficient to demonstrate the generality and practical effectiveness of CERF. The experiments are limited to Fashion-MNIST and FEMNIST with relatively simple CNN models. Although FEMNIST provides a Non-IID setting, the paper does not systematically investigate different levels of data heterogeneity. Furthermore, the experiments assume that clients follow the prescribed participation schedule, without considering client dropout or irregular participation. The additional memory and computational overhead introduced by residual storage and feedback is also not adequately evaluated. More comprehensive experiments on realistic federated learning scenarios are needed to support the claimed benefits.

Questions for authors

Under the random allocation scheme, clients independently select their participation rounds, so the number of participating clients in each round may vary. However, the algorithm and privacy analysis use a fixed value of r = n/K. How is the aggregation and noise calibration handled when the actual number of participating clients differs from r, and does the stated privacy guarantee still hold?

Reviewer_ZJJCOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the gradient sparsification with error feedback in federated learning. Two problems are solved. The first is the noise in stored gradient and the increasement of the gradient norm. To solve this, the proposed CERF method only adds stored gradient at masked position, and stores unmasked gradient before adding gaussian noise. The second problem is the staled update. To solve this, CERF forces exactly k update of the K rounds for each client.

Critical strengths

A clear pipeline of CERF is given.

Theoretical analysis is provided for the proposed CERF method.

Critical weaknesses
  • I didn't see the key difference of the proposed CERF method compared with previous works such as [1][2].
  • For Random-k Sparsifier, previous methods such as [1] have already studied the random sparsification.
  • For Random Allocation, i.e., forcing exactly k update of the K rounds for each client, this is not technical method to solve the staleness problem, just sidesteps it. This forced update may be impractical in real-world use.
  • The problem studied is evaluated only on small-scale and toy benchmarks, and I did not recognize its key usefulness in the era of large language models.
  • The comparison with SOTA methods is limited.

Reference: [1] Gradient Sparsification for Efficient Wireless Federated Learning with Differential Privacy. [2] FedSparQ: Adaptive Sparse Quantization with Error Feedback for Robust & Efficient Federated Learning.

Questions for authors

My questions for the authors to rebutal are the same with that in the Weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.