- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes FedCADC (Federated Learning with Curvature-Aware Direction Consistency), a federated learning framework designed to improve global generalization by seeking flatter global minima under heterogeneous client data. The authors argue that existing sharpness-aware federated learning methods primarily focus on local flatness, which does not necessarily translate into global flatness after model aggregation due to inconsistent local curvature and optimization directions.
To address this issue, FedCADC introduces two regularization terms into local optimization: (1) a curvature-aware regularizer that penalizes perturbation-induced loss increases and (2) a direction-consistency regularizer that encourages local update trajectories to align with the historical global optimization direction. The strengths of both regularizers are adaptively adjusted based on mini-batch gradient geometry and directional disagreement. The paper also provides theoretical analyses of optimization convergence, generalization, and local–global flatness discrepancy.
Experiments on CIFAR-10 and CIFAR-100 under various non-IID settings demonstrate that FedCADC achieves competitive or superior classification accuracy compared to existing federated learning methods. The authors further evaluate the flatness of the learned global models using loss variation, Hessian eigenvalues, and local–global flatness discrepancy.
Critical strengths
- Adaptive regularization based on optimization geometry
FedCADC adaptively adjusts the strengths of its curvature-aware and direction-consistency regularizers based on the current optimization state. In particular, the curvature-aware coefficient considers changes in gradient direction and magnitude induced by perturbations, while the direction-consistency coefficient depends on the angular deviation between local and historical global update directions. This adaptive mechanism provides flexibility in controlling the influence of the two regularizers, potentially reducing the need for manually selecting fixed regularization strengths across heterogeneous settings.
- Extensive empirical comparisons
The paper provides extensive experimental comparisons with a broad range of federated learning baselines, including recent flatness-aware methods such as FedSMOO, FedGF, FedGloSS, FedVSSAM, and FedScar. Experiments cover CIFAR-10 and CIFAR-100 with different levels of data heterogeneity and client participation. Beyond classification accuracy, the authors evaluate global loss landscape properties using multiple flatness metrics, including mean absolute loss variation, the largest Hessian eigenvalue, and local–global flatness discrepancy. These evaluations provide empirical support for the effectiveness of the proposed framework.
Critical weaknesses
Q1: Limited novelty of curvature-aware regularization
The proposed curvature-aware regularizer appears closely related to the standard SAM objective. Specifically, the resulting gradient can be written as , which corresponds to an adaptive interpolation between the center and perturbed gradients. Since SAM itself can be expressed as , the proposed component seems to primarily introduce adaptive reweighting rather than a fundamentally different curvature-aware optimization mechanism. The authors should clarify its conceptual novelty and provide comparisons against simpler adaptive or weighted SAM variants.
Q2: Unclear relationship between direction consistency and global flatness
The global direction-consistency regularizer encourages local updates to align with the historical global optimization direction. However, directional alignment and global flatness are fundamentally different properties. Aligning local trajectories with a common descent direction does not necessarily reduce the curvature or sharpness of the global loss landscape. Although the theoretical analysis provides conditional bounds on client-update deviation and local–global flatness discrepancy, these results do not directly establish that directional alignment leads to flatter global minima. A stronger theoretical justification or targeted empirical analysis is needed to substantiate this central claim.
Questions for authors
Minor Questions
- Notation clarification: Please explicitly define the notation in Eq. (12).
- Relationship to FedGF: FedGF, which is cited in this paper, introduces a flatness discrepancy measure in Eq. (7). Could the authors clarify how the flatness discrepancy defined in this paper relates to that of FedGF, particularly whether the two measures are equivalent?
14 comments
Response to Critical Weakness 2 / Question 2 (Reviewer d2C6) — Part 1: Theory
We thank the reviewer for identifying the need to directly evaluate global flatness. We have added targeted ablation experiments in Appendix E.7, “Effect of Direction Consistency on Global Flatness,” including roundwise flatness measurements and global-model loss landscapes. These experiments examine whether adding direction consistency to curvature-aware optimization improves the geometry of the resulting global models.
We have also strengthened the theoretical analysis in Appendix A.4. Proposition 5 and Eq. (67) give a direct upper bound on global sharpness at the communication endpoint that includes a term controlled by client-update deviation. Under the geometric-alignment condition, contraction of this deviation tightens that term when the other terms are held fixed. This provides a theoretical link between direction consistency and global sharpness, complemented by the direct measurements in Appendix E.7.
A direct global-sharpness bound. Proposition 5 establishes
\mathbb{E}\mathcal{S}_F (w^{t+1},b_g^{t+1};\rho) &\le \mathbb{E}\sum_i p_i \mathcal{S}_i(w_i^{t,K},b_i^{t,K};\rho) +L_{b,\rho}\beta_t \\\\ &\quad+ L\rho\sqrt{ \nu_t^2+ \eta^2K^2 \left( \sqrt{D_t^2} +\frac{\sigma}{\sqrt K} \right)^2 }. \end{aligned}$$ The first term captures local sharpness, while $\beta_t$ accounts for inference-state mismatch and $\nu_t^2$ for partial-participation endpoint error. The client-update deviation $D_t^2$ enters the bound through the dispersion of client endpoints. When the geometric-alignment condition holds across the local steps, contraction of $D_t^2$ tightens this dispersion-dependent term, with the other terms held fixed. In response to Reviewer oUVs’s request to verify that condition, we added Section 6.2.5 and Figures 3–4. The paired-trajectory measurements show that geometric alignment holds across the evaluated CIFAR-10 and CIFAR-100 heterogeneity settings, providing empirical support for the condition used in this theoretical link.Response to Critical Weakness 2 / Question 2 (Reviewer d2C6) — Part 2: Empirical Evidence
This is the second part of our response to Critical Weakness 2 / Question 2. We now report the targeted comparisons that directly assess the resulting global models.
Targeted comparison isolating direction consistency. We compare complete FedCADC with FedCADC w/o , retaining the curvature-aware term in both variants. The experiments cover CIFAR-10 and CIFAR-100 under , with ten participating clients per round. This comparison directly tests the contribution of direction consistency within the proposed algorithm, with curvature-aware optimization present in both methods.
Direct measurements of global curvature and sharpness. Figures 10 and 11 report MALV, the maximum Hessian eigenvalue , global sharpness, and the local–global flatness discrepancy throughout training. Faint curves show roundwise measurements, while prominent curves show fitted trends. Measurements start at round two, when a historical global update direction becomes available.
Across both datasets, complete FedCADC generally exhibits lower values of these metrics than FedCADC w/o over substantial portions of training. The metrics also show an overall downward trend. Crucially, compared with the variant without direction consistency, FedCADC achieves lower global curvature and sharpness, extending beyond a reduction in local–global flatness discrepancy. These observations support the contribution of direction consistency to flatter global solutions when combined with curvature-aware optimization.
Global loss-landscape evidence. Figure 12 compares the global-model loss landscapes on CIFAR-10 under the three Dirichlet settings. Within the visualized perturbation planes, FedCADC generally exhibits broader low-loss regions at comparable loss levels than its variant without direction consistency. This indicates reduced sensitivity to parameter perturbations and provides visual evidence consistent with the quantitative curvature and sharpness measurements.
Connection to the theoretical mechanism. Theorem 1 establishes client-update contraction under the geometric-alignment condition, while Theorem 4 connects client-update deviation to the local–global flatness-discrepancy bound. Following Reviewer oUVs’s request for empirical verification of geometric alignment, we added the experiments in Section 6.2.5 and Figures 3–4 of the revised manuscript. These experiments verify that the geometric-alignment condition holds in the evaluated heterogeneous settings. Combined with the direction-consistency ablations in Appendix E.7, these results provide complementary evidence that direction consistency reduces client-update deviation and, when integrated with curvature-aware optimization, helps FedCADC achieve lower global curvature and sharpness and obtain flatter global solutions.
Changes and location. Appendix A.4, Proposition 5 and Eq. (67): a new global-sharpness bound at the communication endpoint, its proof, and analysis of the client-update deviation term. Section 6.2.5 and Figures 3–4: empirical verification of the geometric-alignment condition used in the theoretical analysis. Appendix E.7, Figures 10–11: roundwise global curvature and sharpness comparisons with and without direction consistency. Appendix E.7, Figure 12: global loss-landscape comparisons.
Response to Critical Weakness 1 / Question 1 (Reviewer d2C6)
We thank the reviewer for identifying this important connection. We agree with the algebraic observation and have clarified the contribution accordingly. The curvature-aware component introduces a geometry-dependent rule for controlling the perturbation-induced correction within the center–perturbed gradient family. To establish the value of this design, we have added direct comparisons against simpler adaptive and fixed-weight variants in Appendix E.6, Table 16 and Figure 9.
Relationship to SAM and scope of the contribution. With the perturbation and adaptive coefficient held fixed during differentiation, and when the perturbed loss exceeds the center loss, the center-loss and curvature terms yield
\lambda=\frac{\alpha_{\mathrm{curv}}}{\rho^2}.$$ Thus, $\lambda=1$ recovers the SAM gradient $g_p$. We explicitly state this relationship in the revised manuscript. Our contribution is the choice of a locally adaptive correction strength based jointly on the directional and relative magnitude responses of the gradient, together with its integration into globally coordinated federated optimization. **Why use the joint geometric response?** Writing $c=\cos(g_p,g_c)$ and $q=\\|g_p\\|/\\|g_c\\|$, Proposition 2 gives, without numerical stabilization, $$\\|g_p-g_c\\|^2 =\\|g_c\\|^2\bigl[(q-1)^2+2q(1-c)\bigr].$$ This identity distinguishes changes in gradient magnitude from changes in gradient direction. Motivated by their complementary information, Eq. (13) combines $1-c$ with $|\log q|$, using numerical stabilization and clipping in the implementation. The product attenuates the weight when either signal is small and emphasizes perturbations that substantially change both. The logarithmic factor also treats reciprocal norm increases and decreases symmetrically. This supplies a specific geometric criterion for adapting the correction strength; its empirical value is tested by the following controlled comparisons. **Comparisons against simpler adaptive and weighted variants.** Table 16 compares joint weighting with gradient-norm-only weighting, gradient-direction-only weighting, and fixed correction weights $\lambda\in\\{0.1,0.5,1.0,1.5,2.0\\}$. All variants retain the same global direction-consistency module, so the comparison isolates the curvature-weighting strategy. The $\lambda=1$ variant provides the SAM-gradient correction with the same direction-consistency module. Experiments cover CIFAR-10 and CIFAR-100 under three Dirichlet settings, with ten participating clients per round. We report means and standard deviations across five independent runs. FedCADC achieves the highest mean accuracy and lowest standard deviation in all six settings in Table 16. Its accuracy exceeds the strongest alternative in each setting by $0.55$–$6.55$ percentage points. For example, on CIFAR-10 with $\alpha_{\mathrm{Dir}}=0.1$, joint weighting achieves $73.86\\%$, compared with $67.31\\%$ for the best fixed-weight variant, $65.72\\%$ for direction-only weighting, and $61.62\\%$ for norm-only weighting. Figure 9 further illustrates the convergence behavior. Because global direction consistency is retained throughout, these gains provide direct evidence for the value of the joint weighting rule beyond simply adding global guidance to SAM. **Changes and location.** Appendix E.6, “Ablation of Curvature-Aware Weighting,” Table 16 and Figure 9: explicit clarification of the SAM relationship, geometric motivation, and comparisons against simpler adaptive and fixed-weight variants. The revised discussions and newly added passages are highlighted in blue.