acceptodds
Withdrawn from ICLR 2027

An Adaptive Penalty Method for Heterogeneous Federated Learning with Nonlinear Constraints

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Federated learning (FL) enables collaborative model training without sharing raw data and is typically formulated as unconstrained minimization of a global loss. To the best of our knowledge, no existing methods provide provable convergence guarantees for nonlinearly constrained FL while explicitly mitigating client drift. We propose an adaptive-penalty Moreau-smoothed stochastic controlled federated averaging method that extends SCAFFOLD to FL with both local and global nonlinear constraints. The penalty parameter and stepsize are updated at each communication round according to analytically derived rules. In the convex setting, we achieve communication complexity for both global suboptimality and constraint violation. In the strongly convex setting, the complexity for both quantities improves to . Some of the unique features of our method include the following: (i) In both regimes, the global suboptimality complexity matches the -dependence of SCAFFOLD and FedAvg in the unconstrained setting; (ii) unlike limited existing constrained FL analyses that rely on Lagrangian duality and constraint qualifications, our penalty-based analysis establishes upper bounds on both global suboptimality and constraint violation without requiring constraint qualifications. We show that a constraint qualification is needed only to establish the lower bound on global suboptimality; and (iii) under Slater's condition, we show that the expected suboptimality and infeasibility errors converge asymptotically to zero for any arbitrary Moreau smoothing parameter. Numerical experiments on MNIST, EMNIST, and CIFAR-10 show that the proposed method attains competitive performance while consistently reducing constraint violation.

Primary area:
optimization
Keywords:
Federated Learning, Constrained Optimization, Client Drift, Moreau Smoothing, Adaptive Penalty Method, Communication Complexity, Convergence Guarantees

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_tcoSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies federated learning with nonlinear local and global functional constraints (Eq. 1), a setting where existing constrained-FL analyses rely on Lagrangian duality, constraint qualifications, or full participation. It proposes AdaP-SCAFFOLD, which extends SCAFFOLD's control variates with a round-varying penalty parameter and local stepsize updated by analytically derived rules (Algorithm 1). Constraint enforcement uses an eta-Moreau-smoothed ReLU penalty that grows linearly in the infeasible region, preserves the feasible set exactly, and admits explicit local Lipschitz constants (Definition 3.1 and Proposition 3.6). In the convex setting the method achieves O(eps^-2) communication complexity for both suboptimality and constraint violation, improving to O(eps^-1) under strong convexity, matching the eps-dependence of unconstrained SCAFFOLD (Theorems 4.1 and 4.3), and the upper bounds hold without constraint qualifications. Experiments on MNIST, EMNIST, and CIFAR-10 compare against ProxAL and unconstrained FL baselines, reporting competitive accuracy with consistently lower constraint violation.

Critical strengths
  1. The problem coverage is a genuine step forward for constrained FL. Handling client-specific and shared functional constraints together with stochastic gradients, partial participation, and drift correction covers a combination that Table 1 shows prior work does not, and obtaining upper bounds on both suboptimality and infeasibility without constraint qualifications is a real theoretical convenience.
  2. The Moreau-smoothed penalty is a well-chosen technical device. Proposition 3.6 shows it preserves the exact feasible set while restoring global smoothness, and this translates into visible empirical behavior, for example on EMNIST the average violation drops from about 0.095 for ProxAL to 0.008 with feasible ratio rising from 0 to 0.67 (Table 2).
Critical weaknesses
  1. The experiments do not test the paper's central algorithmic claim. The single constrained-FL competitor, ProxAL, is evaluated under its own full-participation full-gradient regime rather than the partial-participation stochastic setting the theory targets, the unconstrained baselines in Figures 2 and 3 cannot satisfy constraints by construction so the comparison is expected, and no ablation isolates the adaptive penalty and stepsize rules against a fixed well-tuned penalty, which is the core design contribution.
  2. There is an unaddressed theory-practice gap and the rates are weak in the constants. The guarantees assume convex objectives and constraints, yet all experiments train neural networks, and the bounds carry factors like exp(448 c_l^2 L_f^2) in Theorem 4.1, so the matched eps-dependence is meaningful only as an asymptotic statement. Absolute accuracy is also low (about 0.40 on CIFAR-10), leaving practical relevance undemonstrated.
Questions for authors
  1. The experiments do not test the paper's central algorithmic claim. The single constrained-FL competitor, ProxAL, is evaluated under its own full-participation full-gradient regime rather than the partial-participation stochastic setting the theory targets, the unconstrained baselines in Figures 2 and 3 cannot satisfy constraints by construction so the comparison is expected, and no ablation isolates the adaptive penalty and stepsize rules against a fixed well-tuned penalty, which is the core design contribution.
  2. There is an unaddressed theory-practice gap and the rates are weak in the constants. The guarantees assume convex objectives and constraints, yet all experiments train neural networks, and the bounds carry factors like exp(448 c_l^2 L_f^2) in Theorem 4.1, so the matched eps-dependence is meaningful only as an asymptotic statement. Absolute accuracy is also low (about 0.40 on CIFAR-10), leaving practical relevance undemonstrated.
Reviewer_EqFPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Existing methods lack provable convergence guarantees for federated learning with nonlinear constraints while explicitly mitigating client drift. To address this limitation, this paper proposes an adaptive-penalty Moreau-smoothed stochastic controlled federated averaging method that extends SCAFFOLD to federated learning with both local and global nonlinear constraints. Numerical experiments on MNIST, EMNIST, and CIFAR-10 demonstrate the effectiveness of the proposed method.

Critical strengths
  1. The paper proposes a joint adaptive adjustment mechanism for the penalty parameter and learning rate, providing a theoretical foundation for algorithm convergence.

  2. The paper replaces the traditional quadratic ReLU penalty with a Moreau-smoothed ReLU penalty function, ensuring that the constraint penalty function is continuously differentiable and has Lipschitz-continuous gradients under the corresponding assumptions.

  3. In convex and strongly convex optimization settings, the algorithm achieves communication complexities of O~(ϵ−2)\widetilde{O}(\epsilon^{-2}) and O~(ϵ−1)\widetilde{O}(\epsilon^{-1}), respectively, for both global suboptimality and constraint violation.

  4. The paper proposes AdaP-SCAFFOLD, which integrates the control variate mechanism with nonlinear constraint penalties, enabling the simultaneous handling of client-specific constraints and globally shared constraints while addressing client data heterogeneity.

Critical weaknesses
  1. All experiments in this paper employ a single-layer linear model to remain consistent with the convex optimization setting assumed in the theoretical analysis. Although this choice facilitates the validation of the theoretical results, it remains unclear whether the proposed method can maintain similar performance advantages in more complex nonconvex models.

  2. The experiments involve only five clients, with three selected to participate in each communication round, for a total of 200 communication rounds. This setting does not adequately reflect the communication overhead, control variate maintenance costs, and imbalanced client participation that may arise in scenarios involving hundreds of clients.

  3. Equation (1) considers both client-specific local constraints hi(x)≤0h_i(x)\leq 0 and a global constraint h0(x)≤0h_0(x)\leq 0. However, the main evaluation metrics in Appendix A.5 primarily focus on the average violation and feasible ratio of client-specific local constraints. The paper does not sufficiently report global constraint violations separately, nor does it provide experiments investigating different degrees of conflict between local and global constraints.

Questions for authors
  1. All experiments in this paper employ a single-layer linear model to remain consistent with the convex optimization setting assumed in the theoretical analysis. Although this choice facilitates the validation of the theoretical results, it remains unclear whether the proposed method can maintain similar performance advantages in more complex nonconvex models.

  2. The experiments involve only five clients, with three selected to participate in each communication round, for a total of 200 communication rounds. This setting does not adequately reflect the communication overhead, control variate maintenance costs, and imbalanced client participation that may arise in scenarios involving hundreds of clients.

  3. Equation (1) considers both client-specific local constraints hi(x)≤0h_i(x)\leq 0 and a global constraint h0(x)≤0h_0(x)\leq 0. However, the main evaluation metrics in Appendix A.5 primarily focus on the average violation and feasible ratio of client-specific local constraints. The paper does not sufficiently report global constraint violations separately, nor does it provide experiments investigating different degrees of conflict between local and global constraints.

  4. As shown in Table 2, the proposed method does not outperform existing methods in terms of constraint violation under certain experimental settings, and the number of independent experimental runs is relatively limited.

Reviewer_B5CWOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper focuses on the method to provide provable convergence guarantees for nonlinearly constrained FL while explicitly mitigating client drift. The proposed adaptive-penalty Moreau-smoothed stochastic controlled method extends SCAFFOLD, considers both local and global nonlinear constraints, and updates the penalty parameter and stepsize at each communication round. They provide theoretical analysis where the algorithm achieves O(ϵ−2)O(\epsilon^{−2}) communication complexity in the convex setting and O(ϵ−1)O(\epsilon^{−1}) in the strongly convex setting. Their penalty-based analysis establishes upper bounds on both global suboptimality and constraint violation without requiring constraint qualifications.

Critical strengths
  1. This paper consider the both local and global nonlinear constraints, which is better than the requirement for the average constraint across clients.

  2. The provided convergence bound is promising and provides non-asymptotic guarantees under weak assumptions.

Critical weaknesses
  1. The application is limited. This paper only considers the convex setting and the strongly convex setting. However, ALL neural networks are non-convex functions. The application of a convex objective is very limited.

  2. Assumption 2.1. requires " set hi(x)≤0hi(x) ≤ 0 is bounded, it means each individual constraint set to be bounded. This is a stronger assumption than requiring their intersection to be bounded, and it excludes individual affine inequalities as well as many general functional constraints.

  3. Ablation study is missing. We don't know the method sensitivity to the parameters. We also don't know what if add directly add the penalty to the SCAFFOLD.

  4. Some statements are unclear. For example.

  1. "client participation fraction 0.7 (3 out of 5 clients per round)," 3 out 5 should be 0.6;

  2. It defined the smoothed penalty function gi,η(x) ≜ rη(hi(x)). But the function max{hi(ˆxR+1), 0} in the experiment is not smooth.

  1. The proof is incorrect.
  1. Line 950, lemma 4 mismatch and C_x is missing.

  2. line 1592. second inequality is invalid. For m=1, we can get 2 < (2)\sqrt(2)

Questions for authors

Seen weakness.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.