acceptodds
Under review as a conference paper at ICLR 2027

FedCADC: Curvature-Aware Direction Consistency for Globally Flat Federated Learning

Abstract

Despite the success of federated learning (FL), improving global generalization under heterogeneous client data remains challenging. To improve the generalization of FL, recent methods seek flatter minima by applying sharpness-aware optimization during local training. However, differences in local curvature and update directions can make locally flat solutions inconsistent after aggregation, such that local flatness does not necessarily translate into global flatness. To address this issue, we propose Federated Learning with Curvature-Aware Direction Consistency (FedCADC). FedCADC uses a curvature-aware constraint to suppress local updates toward perturbation-sensitive regions and a direction-consistency constraint to align local trajectories with the global optimization trend. Their weights are adaptively adjusted according to the optimization geometry of each mini-batch. Our theoretical analysis characterizes the optimization, generalization, and flatness discrepancy of FedCADC. Experiments on CIFAR-10 and CIFAR-100 demonstrate that FedCADC achieves higher accuracy in most heterogeneous settings and generally finds flatter global solutions than recent advanced FL methods.

Primary area:
optimization
Keywords:
Federated Learning, Data Heterogeneity, Sharpness-Aware Optimization, Flat Minima, Direction Consistency

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_d2C6OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes FedCADC (Federated Learning with Curvature-Aware Direction Consistency), a federated learning framework designed to improve global generalization by seeking flatter global minima under heterogeneous client data. The authors argue that existing sharpness-aware federated learning methods primarily focus on local flatness, which does not necessarily translate into global flatness after model aggregation due to inconsistent local curvature and optimization directions.

To address this issue, FedCADC introduces two regularization terms into local optimization: (1) a curvature-aware regularizer that penalizes perturbation-induced loss increases and (2) a direction-consistency regularizer that encourages local update trajectories to align with the historical global optimization direction. The strengths of both regularizers are adaptively adjusted based on mini-batch gradient geometry and directional disagreement. The paper also provides theoretical analyses of optimization convergence, generalization, and local–global flatness discrepancy.

Experiments on CIFAR-10 and CIFAR-100 under various non-IID settings demonstrate that FedCADC achieves competitive or superior classification accuracy compared to existing federated learning methods. The authors further evaluate the flatness of the learned global models using loss variation, Hessian eigenvalues, and local–global flatness discrepancy.

Critical strengths
  • Adaptive regularization based on optimization geometry

FedCADC adaptively adjusts the strengths of its curvature-aware and direction-consistency regularizers based on the current optimization state. In particular, the curvature-aware coefficient considers changes in gradient direction and magnitude induced by perturbations, while the direction-consistency coefficient depends on the angular deviation between local and historical global update directions. This adaptive mechanism provides flexibility in controlling the influence of the two regularizers, potentially reducing the need for manually selecting fixed regularization strengths across heterogeneous settings.

  • Extensive empirical comparisons

The paper provides extensive experimental comparisons with a broad range of federated learning baselines, including recent flatness-aware methods such as FedSMOO, FedGF, FedGloSS, FedVSSAM, and FedScar. Experiments cover CIFAR-10 and CIFAR-100 with different levels of data heterogeneity and client participation. Beyond classification accuracy, the authors evaluate global loss landscape properties using multiple flatness metrics, including mean absolute loss variation, the largest Hessian eigenvalue, and local–global flatness discrepancy. These evaluations provide empirical support for the effectiveness of the proposed framework.

Critical weaknesses

Q1: Limited novelty of curvature-aware regularization

The proposed curvature-aware regularizer appears closely related to the standard SAM objective. Specifically, the resulting gradient can be written as gc+λ(gp−gc)g_c+\lambda(g_p-g_c), which corresponds to an adaptive interpolation between the center and perturbed gradients. Since SAM itself can be expressed as gc+(gp−gc)g_c+(g_p-g_c), the proposed component seems to primarily introduce adaptive reweighting rather than a fundamentally different curvature-aware optimization mechanism. The authors should clarify its conceptual novelty and provide comparisons against simpler adaptive or weighted SAM variants.

Q2: Unclear relationship between direction consistency and global flatness

The global direction-consistency regularizer encourages local updates to align with the historical global optimization direction. However, directional alignment and global flatness are fundamentally different properties. Aligning local trajectories with a common descent direction does not necessarily reduce the curvature or sharpness of the global loss landscape. Although the theoretical analysis provides conditional bounds on client-update deviation and local–global flatness discrepancy, these results do not directly establish that directional alignment leads to flatter global minima. A stronger theoretical justification or targeted empirical analysis is needed to substantiate this central claim.

Questions for authors

Minor Questions

  1. Notation clarification: Please explicitly define the notation ⋅+\cdot_+ in Eq. (12).
  2. Relationship to FedGF: FedGF, which is cited in this paper, introduces a flatness discrepancy measure in Eq. (7). Could the authors clarify how the flatness discrepancy defined in this paper relates to that of FedGF, particularly whether the two measures are equivalent?
14 comments
AuthorsOpenReview ↗

Response to Critical Weakness 1 / Question 1 (Reviewer d2C6)

We thank the reviewer for identifying this important connection. We agree with the algebraic observation and have clarified the contribution accordingly. The curvature-aware component introduces a geometry-dependent rule for controlling the perturbation-induced correction within the center–perturbed gradient family. To establish the value of this design, we have added direct comparisons against simpler adaptive and fixed-weight variants in Appendix E.6, Table 16 and Figure 9.

Relationship to SAM and scope of the contribution. With the perturbation and adaptive coefficient held fixed during differentiation, and when the perturbed loss exceeds the center loss, the center-loss and curvature terms yield

\lambda=\frac{\alpha_{\mathrm{curv}}}{\rho^2}.$$ Thus, $\lambda=1$ recovers the SAM gradient $g_p$. We explicitly state this relationship in the revised manuscript. Our contribution is the choice of a locally adaptive correction strength based jointly on the directional and relative magnitude responses of the gradient, together with its integration into globally coordinated federated optimization. **Why use the joint geometric response?** Writing $c=\cos(g_p,g_c)$ and $q=\\|g_p\\|/\\|g_c\\|$, Proposition 2 gives, without numerical stabilization, $$\\|g_p-g_c\\|^2 =\\|g_c\\|^2\bigl[(q-1)^2+2q(1-c)\bigr].$$ This identity distinguishes changes in gradient magnitude from changes in gradient direction. Motivated by their complementary information, Eq. (13) combines $1-c$ with $|\log q|$, using numerical stabilization and clipping in the implementation. The product attenuates the weight when either signal is small and emphasizes perturbations that substantially change both. The logarithmic factor also treats reciprocal norm increases and decreases symmetrically. This supplies a specific geometric criterion for adapting the correction strength; its empirical value is tested by the following controlled comparisons. **Comparisons against simpler adaptive and weighted variants.** Table 16 compares joint weighting with gradient-norm-only weighting, gradient-direction-only weighting, and fixed correction weights $\lambda\in\\{0.1,0.5,1.0,1.5,2.0\\}$. All variants retain the same global direction-consistency module, so the comparison isolates the curvature-weighting strategy. The $\lambda=1$ variant provides the SAM-gradient correction with the same direction-consistency module. Experiments cover CIFAR-10 and CIFAR-100 under three Dirichlet settings, with ten participating clients per round. We report means and standard deviations across five independent runs. FedCADC achieves the highest mean accuracy and lowest standard deviation in all six settings in Table 16. Its accuracy exceeds the strongest alternative in each setting by $0.55$–$6.55$ percentage points. For example, on CIFAR-10 with $\alpha_{\mathrm{Dir}}=0.1$, joint weighting achieves $73.86\\%$, compared with $67.31\\%$ for the best fixed-weight variant, $65.72\\%$ for direction-only weighting, and $61.62\\%$ for norm-only weighting. Figure 9 further illustrates the convergence behavior. Because global direction consistency is retained throughout, these gains provide direct evidence for the value of the joint weighting rule beyond simply adding global guidance to SAM. **Changes and location.** Appendix E.6, “Ablation of Curvature-Aware Weighting,” Table 16 and Figure 9: explicit clarification of the SAM relationship, geometric motivation, and comparisons against simpler adaptive and fixed-weight variants. The revised discussions and newly added passages are highlighted in blue.
AuthorsOpenReview ↗

Response to Critical Weakness 2 / Question 2 (Reviewer d2C6) — Part 1: Theory

We thank the reviewer for identifying the need to directly evaluate global flatness. We have added targeted ablation experiments in Appendix E.7, “Effect of Direction Consistency on Global Flatness,” including roundwise flatness measurements and global-model loss landscapes. These experiments examine whether adding direction consistency to curvature-aware optimization improves the geometry of the resulting global models.

We have also strengthened the theoretical analysis in Appendix A.4. Proposition 5 and Eq. (67) give a direct upper bound on global sharpness at the communication endpoint that includes a term controlled by client-update deviation. Under the geometric-alignment condition, contraction of this deviation tightens that term when the other terms are held fixed. This provides a theoretical link between direction consistency and global sharpness, complemented by the direct measurements in Appendix E.7.

A direct global-sharpness bound. Proposition 5 establishes

\mathbb{E}\mathcal{S}_F (w^{t+1},b_g^{t+1};\rho) &\le \mathbb{E}\sum_i p_i \mathcal{S}_i(w_i^{t,K},b_i^{t,K};\rho) +L_{b,\rho}\beta_t \\\\ &\quad+ L\rho\sqrt{ \nu_t^2+ \eta^2K^2 \left( \sqrt{D_t^2} +\frac{\sigma}{\sqrt K} \right)^2 }. \end{aligned}$$ The first term captures local sharpness, while $\beta_t$ accounts for inference-state mismatch and $\nu_t^2$ for partial-participation endpoint error. The client-update deviation $D_t^2$ enters the bound through the dispersion of client endpoints. When the geometric-alignment condition holds across the local steps, contraction of $D_t^2$ tightens this dispersion-dependent term, with the other terms held fixed. In response to Reviewer oUVs’s request to verify that condition, we added Section 6.2.5 and Figures 3–4. The paired-trajectory measurements show that geometric alignment holds across the evaluated CIFAR-10 and CIFAR-100 heterogeneity settings, providing empirical support for the condition used in this theoretical link.
AuthorsOpenReview ↗

Response to Critical Weakness 2 / Question 2 (Reviewer d2C6) — Part 2: Empirical Evidence

This is the second part of our response to Critical Weakness 2 / Question 2. We now report the targeted comparisons that directly assess the resulting global models.

Targeted comparison isolating direction consistency. We compare complete FedCADC with FedCADC w/o RdirR_{\mathrm{dir}}, retaining the curvature-aware term in both variants. The experiments cover CIFAR-10 and CIFAR-100 under αDir∈0.1,0.5,10\alpha_{\mathrm{Dir}}\in\\{0.1,0.5,10\\}, with ten participating clients per round. This comparison directly tests the contribution of direction consistency within the proposed algorithm, with curvature-aware optimization present in both methods.

Direct measurements of global curvature and sharpness. Figures 10 and 11 report MALV, the maximum Hessian eigenvalue λmax⁡\lambda_{\max}, global sharpness, and the local–global flatness discrepancy ΔF\Delta_F throughout training. Faint curves show roundwise measurements, while prominent curves show fitted trends. Measurements start at round two, when a historical global update direction becomes available.

Across both datasets, complete FedCADC generally exhibits lower values of these metrics than FedCADC w/o RdirR_{\mathrm{dir}} over substantial portions of training. The metrics also show an overall downward trend. Crucially, compared with the variant without direction consistency, FedCADC achieves lower global curvature and sharpness, extending beyond a reduction in local–global flatness discrepancy. These observations support the contribution of direction consistency to flatter global solutions when combined with curvature-aware optimization.

Global loss-landscape evidence. Figure 12 compares the global-model loss landscapes on CIFAR-10 under the three Dirichlet settings. Within the visualized perturbation planes, FedCADC generally exhibits broader low-loss regions at comparable loss levels than its variant without direction consistency. This indicates reduced sensitivity to parameter perturbations and provides visual evidence consistent with the quantitative curvature and sharpness measurements.

Connection to the theoretical mechanism. Theorem 1 establishes client-update contraction under the geometric-alignment condition, while Theorem 4 connects client-update deviation to the local–global flatness-discrepancy bound. Following Reviewer oUVs’s request for empirical verification of geometric alignment, we added the experiments in Section 6.2.5 and Figures 3–4 of the revised manuscript. These experiments verify that the geometric-alignment condition holds in the evaluated heterogeneous settings. Combined with the direction-consistency ablations in Appendix E.7, these results provide complementary evidence that direction consistency reduces client-update deviation and, when integrated with curvature-aware optimization, helps FedCADC achieve lower global curvature and sharpness and obtain flatter global solutions.

Changes and location. Appendix A.4, Proposition 5 and Eq. (67): a new global-sharpness bound at the communication endpoint, its proof, and analysis of the client-update deviation term. Section 6.2.5 and Figures 3–4: empirical verification of the geometric-alignment condition used in the theoretical analysis. Appendix E.7, Figures 10–11: roundwise global curvature and sharpness comparisons with and without direction consistency. Appendix E.7, Figure 12: global loss-landscape comparisons.

Reviewer_eHEQOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

To address the issue that data heterogeneity can make locally flat solutions inconsistent after aggregation, this paper proposes FedCADC, which combines a curvature-aware constraint with a global directionconsistency constraint and adaptively adjusts their weights according to the mini-batch geometry. Extensive experiments and loss-geometry analyses demonstrate its effectiveness in improving generalization and finding flatter global solutions.

Critical strengths

1.The paper improves global generalization under heterogeneous client data by combining curvature-aware flatness regularization with global direction consistency and adaptively adjusting both constraints using the mini-batch optimization geometry. 2. The paper establishes optimization, generalization, and flatness discrepancy, and evaluates the proposed method against conventional and flatness-driven federated learning baselines under diverse heterogeneous settings. The paper conducts extensive experiments to evaluate the proposed FedCADC from multiple perspectives.

Critical weaknesses
  1. FedCADC relies on specified geometric alignment conditions. It remains unclear how restrictive these assumptions are, whether they are satisfied in practical heterogeneous federated learning scenarios.
  2. The proposed adaptive weighting mechanism is a key component of FedCADC, but its effectiveness and necessity require more convincing empirical validation. For example, Table 3 shows that adaptive adaptive weighting mechanism does not consistently outperform the fixed-coefficient variants.
  3. The proposed method does not consistently outperform the other methods.
Questions for authors
  1. Given that the proposed method does not consistently outperform other methods in terms of test accuracy, does FedCADC offer sufficient overall competitiveness when considering computational overhead, communication cost, convergence time, and model size?
  2. Table 3 shows that adaptive FedCADC does not consistently outperform the fixed-coefficient variants. Why does using a fixed (αcurv=0.01) yield better performance than the adaptive strategy on CIFAR-100? When should fixed coefficients be used, and when should adaptive coefficients be preferred? Furthermore, why are the search ranges for αcurv and αdir set to [0, 0.08] and [0, 0.8], respectively? In particular, what justifies the tenfold difference in the magnitudes of their search ranges? If the adaptive mechanism can dynamically adjust weights, why does a small fixed coefficient (αcurv=0.01) achieve better performance in some experiments?
  3. The computational cost of the proposed FedCADC should be discussed in comparison with that of the baseline methods.
  4. The authors report the standard deviation of the final test accuracy across three independent runs. While this provides an initial assessment of experimental variability, the reliability of the estimated standard deviation may be limited by the small number of runs. The authors are encouraged to consider additional independent runs to better assess the stability of the reported results.
  5. The authors are encouraged to provide sensitivity analyses for key hyperparameters. For example, why is the perturbation radius set to 0.05?
14 comments
AuthorsOpenReview ↗

Response to Critical Weakness

We thank the reviewer for identifying three concerns. We address each below, followed by a detailed response to geometric alignment.

  1. Geometric alignment. Section 6.2.5 and Figures 3–4 test the condition using paired trajectories on CIFAR-10 and CIFAR-100 across three heterogeneity settings. The recorded measurements support client-update contraction; details follow.

  2. Adaptive weighting. Table 3 includes one setting where a fixed coefficient outperforms adaptive FedCADC. The adaptive method leads in five of six settings while reducing setting-specific tuning. Section 6.2.4 discusses the exception, and Appendix B.3 explains the clipping thresholds. See our response to Question 2 from Reviewer eHEQ.

  3. Overall competitiveness. Following the reviewer's suggestion, we increased independent runs from three to five. FedCADC leads in 15/18 Table 1 accuracy settings, versus 12/18 originally; Tables 13–14 report standard deviations. In the main experiments, it has the shortest cumulative computation time in 14/18 target settings and lowest communication cost in 16/18. On the added Yahoo Answers and HuffPost benchmarks, it leads in 5/6 accuracy, 15/18 computation-time, and 17/18 communication-cost settings, while retaining the baselines' backbone model size. See our responses to Questions 1, 3, and 4 from Reviewer eHEQ.

Response to Critical Weakness 1 (Reviewer eHEQ)

We thank the reviewer for asking whether the geometric-alignment condition holds under client heterogeneity. Reviewer oUVs also requested empirical verification. Section 6.2.5, "Empirical Verification of Geometric Alignment," addresses both concerns.

Scope of the condition. Theorem 1 gives a sufficient condition for contraction of the weighted client-update deviations along paired training trajectories. Specifically, the negative cross term between baseline deviations and direction-induced corrections must dominate the positive quadratic correction. This is an aggregate condition: it does not require every client's correction to be negatively aligned with its baseline deviation or require identical client gradients. It therefore permits heterogeneous client behavior when the combined correction reduces weighted update deviation.

Verification under heterogeneous federated learning. Figures 3 and 4 of the revised manuscript report results on CIFAR-10 and CIFAR-100, respectively, for αDir∈0.1,0.5,10\alpha_{\mathrm{Dir}}\in\\{0.1,0.5,10\\} with ten participating clients per round. For each selected client, we compare two local trajectories starting from the same global model and using the same mini-batches. Both retain curvature-aware regularization; one additionally applies direction consistency.

We assess four metrics. The cross term, GAcross\mathrm{GA}_{\mathrm{cross}}, measures the interaction between centered baseline updates and direction-induced corrections; a negative value indicates that the corrections counteract update deviations. The quadratic correction, GAquad\mathrm{GA}_{\mathrm{quad}}, measures the nonnegative squared magnitude of those centered corrections. Contraction occurs when the negative cross term dominates this correction. The normalized contraction margin is GAmargin=−(GAcross+GAquad)/Dbase2\mathrm{GA}_{\mathrm{margin}}=-(\mathrm{GA}_{\mathrm{cross}}+\mathrm{GA}_{\mathrm{quad}})/D_{\mathrm{base}}^2, where Dbase2D_{\mathrm{base}}^2 is the baseline client-update variance; a positive margin indicates relative variance reduction. The deviation ratio, GAratio=Ddir2/Dbase2=1−GAmargin\mathrm{GA}_{\mathrm{ratio}}=D_{\mathrm{dir}}^2/D_{\mathrm{base}}^2=1-\mathrm{GA}_{\mathrm{margin}}, directly compares the variances with and without direction consistency; a value below one indicates contraction.

Across the recorded measurements in all six dataset–heterogeneity combinations, GAcross<0\mathrm{GA}_{\mathrm{cross}} < 0, GAquad>0\mathrm{GA}_{\mathrm{quad}} > 0, GAmargin>0\mathrm{GA}_{\mathrm{margin}} > 0, and GAratio<1\mathrm{GA}_{\mathrm{ratio}} < 1. Thus, the negative cross term dominates the quadratic correction. By the variance identity in Eq. (102), Ddir2<Dbase2D_{\mathrm{dir}}^2 < D_{\mathrm{base}}^2. The normalized margins and deviation ratios also remain relatively stable across the tested heterogeneity levels. These observations show that the condition holds in the evaluated settings, including the strongly heterogeneous case αDir=0.1\alpha_{\mathrm{Dir}}=0.1, and support the contraction mechanism used in the flatness-discrepancy analysis.

Changes and location. Section 6.2.5 and Figures 3–4 contain the new metrics, paired-trajectory experiments, and analysis across heterogeneous settings. The additions are highlighted in blue in the revised manuscript.

AuthorsOpenReview ↗

Response to Question 1 (Reviewer eHEQ)

We thank the reviewer for suggesting a broader assessment of FedCADC's practical competitiveness. We have expanded the analysis of computation, communication, and time to target accuracy in Appendices E.1–E.2. We also evaluate these aspects on the non-visual benchmarks added in response to Reviewer oUVs's request, as reported in Appendix E.8. These results demonstrate FedCADC's overall competitiveness in test accuracy, communication cost, and computation time.

Accuracy and computational overhead. With five independent runs, FedCADC achieves the highest mean accuracy in 15 of the 18 settings in Table 1. Tables 8–9 analyze the theoretical computation costs, while Tables 10–11 report measured client and server times. Measurements use a Windows workstation with an NVIDIA GeForce RTX 5090 D v2 GPU, a 24-core CPU at 3.70 GHz, 64 GB of RAM, and NVMe SSD storage. FedCADC requires two forward–backward passes per local step, as does FedSAM, with approximately 6% additional client-side computation for adaptive weighting and direction consistency. The server performs one model aggregation and an additional O(dpar)\mathcal{O}(d_{\mathrm{par}}) direction update, where dpard_{\mathrm{par}} denotes the number of model parameters.

Computation time to target accuracy. Table 12 shows that FedCADC achieves the shortest cumulative computation time in 14 of the 18 settings. In the 13 winning settings where another method also reaches the target, it reduces time by 1.6%–68.9% relative to the fastest successful baseline. For example, it reaches 70% accuracy on CIFAR-10 with αDir=0.5\alpha_{\mathrm{Dir}}=0.5 in 460.009 seconds, 26.3% faster than FedScar (624.274 seconds). In the remaining winning setting, FedCADC is the only method to reach the target within the training budget.

Communication cost and model size. The curvature-aware term is computed locally and incurs no additional communication cost. Direction consistency requires one additional model-sized download after the first round. For RR rounds with mpartm_{\mathrm{part}} participating clients, FedCADC therefore communicates mpart(3R−1)Smodelm_{\mathrm{part}}(3R-1)S_{\mathrm{model}}. Despite this additional transmission, Table 7 shows the lowest cumulative communication cost in 16 of the 18 target settings. For example, reaching 80% accuracy on CIFAR-10 with αDir=0.5\alpha_{\mathrm{Dir}}=0.5 costs 3193.7 MB, 33.4% less than the best baseline, FedMut. FedCADC uses the same ResNet-20 backbone as the baselines and introduces no additional trainable parameters. Its FP32 model storage is 1.090 MB and 1.113 MB on CIFAR-10 and CIFAR-100, respectively.

Evidence from the non-visual extension. Following Reviewer oUVs's request, we evaluate Yahoo Answers with TextCNN and HuffPost with pretrained BERT-Tiny, whose FP32 parameter storage is 16.16 MB and 17.57 MB, respectively. Table 17 shows that FedCADC achieves the highest mean accuracy in five of six settings, exceeding the strongest baseline in each by 0.47–2.35 percentage points. The only exception is HuffPost at αDir=0.5\alpha_{\mathrm{Dir}}=0.5, where FedCADC achieves 64.56%, only 0.37 percentage points below FedScar's 64.93%. Table 18 shows that direction consistency increases average computation time per round by only 2.78%–3.16%, without additional network forward or backward passes. Table 19 shows that FedCADC achieves the shortest computation time in 15 of 18 target settings, reducing time by 14.0%–29.0% relative to the fastest successful baseline in each of these settings. Table 20 further reports the lowest communication cost in 17 of 18 settings, with savings of 26.1%–49.1% relative to the most communication-efficient successful baseline in each setting. These results demonstrate FedCADC's overall advantages in computation and communication costs across the evaluated tasks, while preserving the backbone model size.

Changes and location. Appendix E.1, Table 7: communication analysis. Appendix E.2, Tables 8–12: theoretical and measured computation costs and time to target accuracy. Appendix E.8, Tables 17–20: non-visual accuracy and efficiency comparisons, with model specifications. The additions and revisions are highlighted in blue.

AuthorsOpenReview ↗

Response to Question 2 (Reviewer eHEQ), Part 1

We thank the reviewer for these questions. The purpose of adaptive weighting is to maintain strong performance across heterogeneous settings while reducing manual coefficient tuning. In the revised manuscript, the main-text ablation discussion acknowledges cases where fixed coefficients perform better. We also distinguish the ablation ranges from the upper bounds used in adaptive weighting.

Why a fixed coefficient can perform better. On CIFAR-100 with αDir=0.5\alpha_{\mathrm{Dir}}=0.5, fixing αcurv=0.01\alpha_{\mathrm{curv}}=0.01 achieves 52.7852.78\\%, compared with 51.7851.78\\% for adaptive FedCADC, a difference of 1.001.00 percentage point. We explicitly acknowledge this result in Section 6.2.4. The adaptive rule responds to local gradient geometry; it does not directly optimize validation accuracy or select the best fixed coefficient for each experimental setting. A constant coefficient can provide a favorable regularization strength for a particular task, potentially avoiding weight fluctuations induced by local geometric signals. However, the best-performing coefficient can vary with the dataset and heterogeneity level. Although αcurv=0.01\alpha_{\mathrm{curv}}=0.01 performs best in this CIFAR-100 setting, it is outperformed by adaptive FedCADC under the other two Dirichlet settings and all three CIFAR-10 settings. The contribution of adaptive weighting is therefore to maintain strong performance across settings while reducing the need for manual, task-specific coefficient tuning.

When to use fixed or adaptive coefficients. Fixed coefficients may be considered when sufficient computational resources are available for validation-based grid search. However, identifying a suitable value requires additional experiments, and a coefficient selected for one setting may not remain optimal when the dataset or client heterogeneity changes. This makes repeated tuning across settings costly. Adaptive weighting reduces this burden by adjusting the coefficients according to the observed local optimization geometry. Our results support its use when strong performance across settings and reduced manual tuning are both priorities.

Ablation ranges and clipping upper bounds. Table 3 evaluates fixed curvature coefficients 0,0.01,0.03,0.05,0.08\\{0,0.01,0.03,0.05,0.08\\} and fixed direction coefficients 0,0.2,0.4,0.6,0.8\\{0,0.2,0.4,0.6,0.8\\}. These grids span zero regularization through the respective upper bounds used by the adaptive mechanism. In Eqs. (13) and (15), Acurv=0.08A_{\mathrm{curv}}=0.08 and Adir=0.8A_{\mathrm{dir}}=0.8 serve as protective clipping limits. They do not multiply or otherwise rescale the underlying geometry-dependent signals; they affect the resulting weights only when those signals exceed the corresponding limits.

The two upper bounds are specified independently according to the distinct scales of their underlying geometric signals. For direction consistency, the unclipped weight is 1−cos⁡(d,gt)1-\cos(d,g^t), which lies in [0,1)[0,1) when the local and historical global directions have positive cosine similarity. We choose Adir=0.8A_{\mathrm{dir}}=0.8 to retain a broad range of geometry-dependent weights: it leaves the weight unchanged for cosine similarities in [0.2,1][0.2,1], covering strongly aligned through weakly positively aligned directions. For cosine similarities below 0.20.2, the weight is capped at 0.80.8 to limit the regularization strength.

Reviewer_oUVsOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Federated Learning with Curvature-Aware Direction Consistency (FedCADC). FedCADC uses a curvature-aware constraint to suppress local updates toward perturbation-sensitive regions and a direction-consistency constraint to align local trajectories with the global optimization trend. The provided theoretical analysis characterizes the optimization, generalization, and flatness discrepancy of FedCADC. Experiments on CIFAR-10 and CIFAR-100 demonstrate that FedCADC achieves higher accuracy in most heterogeneous settings and generally finds flatter global solutions than recent advanced FL methods.

Critical strengths

1.This is a meaningful topic, which attempts to solve an important problem, namely: that local flatness does not transfer to global flatness under client heterogeneity. The motivation of the paper is clear.

  1. The two regular complementary strategies designed in this paper can adaptively reduce the sensitivity of mutual information.

  2. The theoretical results are relatively solid, which includes optimization analysis and generalization analysis, and the assumptions of the theory are reasonable.

  3. The experimental method of comparison is relatively complete, and the proposed algorithm is verified.

Critical weaknesses
  1. The experimental data only use cifar-10/100 with resnet-20. It would be better if the proposed federated algorithm could be applied to large-scale and actual crossover scenarios.
  2. Whether this hypothetical condition in the paper, namely the geometric alignment condition, is in line with the actual situation, it is suggested to discuss it in the paper or verify it with experiments.
  3. The parameters and some computational costs in the experiment need further discussion, such as: it wins 12/18 settings and is slightly worse than FedSMOO on MALV at α_Dir = 0.5 and 10
  4. In the experiment, the hyperparameter settings in the adaptive strategy should also be discussed and reported.
Questions for authors
  1. This article needs to discuss the geometric alignment condition or verify through experiments how this condition promotes the results.
  2. It is best for this paper to provide a broader evaluation scenario, such as a large model, and then report and evaluate the communication costs related to the security aware and direction consistency terms
  3. This paper should clarify whether the theory covers partial client participation and optimizer residuals (e.g., momentum, weight decay, clipping), And verify which techniques work through ablation experiments.
7 comments
AuthorsOpenReview ↗

Response to the Critical Weaknesses

We thank the reviewer for these concerns. We summarize our revisions here and provide the detailed evidence in our responses to the Questions for Authors.

  1. Evaluation scope. We added two non-visual benchmarks: Yahoo Answers with TextCNN and HuffPost with pretrained BERT-Tiny. The revised manuscript reports accuracy, computation time, and communication cost on both tasks. Please see our response to Question 2.

  2. Geometric alignment. We added paired-trajectory experiments in Section 6.2.5 and Figures 3–4. The measured geometric-alignment condition holds across the evaluated CIFAR-10 and CIFAR-100 heterogeneity settings. The theoretical improvement remains conditional on this condition. Please see our response to Question 1.

  3. Performance and overhead. We acknowledge that FedCADC does not lead on every accuracy or flatness metric. The revised manuscript discusses these exceptions and reports component-level computation costs, communication costs, and time to target accuracy for both the main and non-visual experiments. Please see our response to Question 2.

  4. Variability and adaptive hyperparameters. The original submission was already based on three independent runs. Due to space constraints in the layout of Table 1, the corresponding standard deviations were reported separately in Table 7 of Appendix E.3. Following Reviewer eHEQ's suggestion, we have increased the number of independent runs from three to five and updated the reported means and standard deviations. Please see our response to Question 1 for the experimental protocol and revised results.

We also thank the reviewer for noting that the adaptive mechanism introduces hyperparameters whose robustness has been insufficiently explored. To address this concern, we clarify the roles of AcurvA_{\mathrm{curv}} and AdirA_{\mathrm{dir}} in Eqs. (13) and (15), and summarize the supporting adaptive-weight analysis and ablation results.

First, these parameters serve as clipping thresholds rather than fixed regularization strengths. Before clipping, αcurv\alpha_{\mathrm{curv}} is determined by the directional and relative norm changes between the center and perturbed gradients, while αdir\alpha_{\mathrm{dir}} is determined by the angular discrepancy between the local trajectory and the historical global direction. Neither clipping threshold enters these underlying geometric response expressions. Whenever a response lies below its threshold, the corresponding adaptive coefficient is unchanged by the clipping operation and is locally independent of the threshold.

At the same time, in response to Reviewer eHEQ's related question about the choices of these two clipping thresholds, we have expanded Appendix B.3 to explain their theoretical motivation. Specifically, we derive a bound on the unclipped curvature weight and characterize the directional-alignment range covered by the direction-consistency threshold. This analysis, together with the observed scales of the adaptive coefficients, provides a basis for setting Acurv=0.08A_{\mathrm{curv}}=0.08 and Adir=0.8A_{\mathrm{dir}}=0.8 as relatively permissive clipping thresholds in our experiments, allowing the coefficients to adapt over a broad range while limiting excessively large weights. The corresponding explanation is also provided in our response to Question 2 from Reviewer eHEQ.

Figures 10 and 11 in the original submission, corresponding to Figures 16 and 17 in the revised manuscript, report the evolution of the adaptive coefficients αcurv\alpha_{\mathrm{curv}} and αdir\alpha_{\mathrm{dir}} on CIFAR-10 and CIFAR-100 under three Dirichlet settings and with 5, 10, and 20 participating clients per round. The plotted round-averaged coefficients remain well below the clipping thresholds used in our experiments, Acurv=0.08A_{\mathrm{curv}}=0.08 and Adir=0.8A_{\mathrm{dir}}=0.8, while exhibiting changes across training stages and experimental settings. These observations show that the average regularization strengths are not pinned to the prescribed clipping thresholds and retain their adaptive behavior across the evaluated configurations.

Complementary evidence is provided by Table 3, which compares adaptive weighting with fixed curvature coefficients in 0.01,0.03,0.05,0.08\\{0.01,0.03,0.05,0.08\\} and fixed direction-consistency coefficients in 0.2,0.4,0.6,0.8\\{0.2,0.4,0.6,0.8\\}. Adaptive FedCADC achieves the highest mean accuracy in five of the six settings and outperforms every tested fixed-direction-weight variant. Together, the coefficient trajectories and ablation results support the practical value of geometry-dependent weighting in reducing reliance on manually selected regularization strengths across the tested settings.

The corresponding additions and revisions are highlighted in blue in the revised manuscript.

AuthorsOpenReview ↗

Response to Question 1 (Reviewer oUVs)

We thank the reviewer for requesting empirical verification of the geometric-alignment condition and clearer reporting of results across independent random seeds. We address these two points below.

Empirical verification of geometric alignment. We have added Section 6.2.5, “Empirical Verification of Geometric Alignment,” with Figures 3 and 4 presenting the results on CIFAR-10 and CIFAR-100, respectively. The experiments cover αDir∈0.1,0.5,10\alpha_{\mathrm{Dir}} \in \\{0.1,0.5,10\\} with ten participating clients per round. We compare paired local trajectories that both retain the curvature-aware term, with direction consistency additionally applied to one trajectory. Both trajectories start from the same global model and use the same mini-batches.

Four metrics characterize geometric alignment. The cross term (GAcross\mathrm{GA}_{\mathrm{cross}}) measures the interaction between centered baseline updates and direction-induced corrections; negative values indicate that the corrections counteract client-update deviations. The quadratic correction (GAquad\mathrm{GA}_{\mathrm{quad}}) is the nonnegative squared magnitude of the centered corrections, which must be outweighed by the negative cross term for contraction to occur. The normalized contraction margin is defined as

= -\frac{\mathrm{GA}_{\mathrm{cross}}+\mathrm{GA}_{\mathrm{quad}}} {D_{\mathrm{base}}^2},$$ where $D_{\mathrm{base}}^2$ denotes the baseline client-update variance. A positive margin indicates a relative reduction in client-update variance. The client-update deviation ratio ($\mathrm{GA}_{\mathrm{ratio}}$) compares the variance with direction consistency to that of the baseline. The two metrics satisfy $\mathrm{GA}_{\mathrm{ratio}}=1-\mathrm{GA}_{\mathrm{margin}}$; thus, a positive margin is equivalent to a ratio below one. Across the evaluated settings, all recorded values satisfy $\mathrm{GA}_{\mathrm{cross}} < 0$, $\mathrm{GA}_{\mathrm{quad}} > 0$, $\mathrm{GA}_{\mathrm{margin}} > 0$, and $\mathrm{GA}_{\mathrm{ratio}} < 1$. The negative cross term therefore dominates the positive quadratic correction in these experiments. By the variance identity in Eq. (102), this gives $D_{\mathrm{dir}}^2 < D_{\mathrm{base}}^2$, providing empirical support for the sufficient condition in Theorem 1 and the client-update contraction mechanism associated with the flatness-discrepancy analysis in Theorem 4. The theoretical improvement remains conditional on geometric alignment; these experiments show that the condition holds across the evaluated heterogeneity settings. **Multi-seed means and standard deviations.** We appreciate the reviewer's concern about experimental variability. The original submission was already based on three independent runs. Due to space constraints in the layout of Table 1, the corresponding standard deviations were reported separately in Table 7 of Appendix E.3. We recognize that separating the means and standard deviations may have made their correspondence less apparent, and have clarified it in the revised manuscript. Furthermore, following Reviewer eHEQ's suggestion, we increased the number of independent runs from three to five. For each run, final test accuracy is obtained by averaging the last 50 communication rounds. The reported mean and standard deviation are then computed across the five run-level averages. With the updated five-run results, FedCADC achieves the highest mean accuracy in 15 of the 18 settings in Table 1, compared with 12 of the 18 settings in the original three-run Table 1. In the revised manuscript, Tables 1 and 3 report the mean accuracies for the algorithm comparisons and adaptive-weight ablations, respectively; Tables 13 and 14 provide their corresponding standard deviations in percentage points. Appendix E.4, “Standard Deviations of Test Accuracy in Algorithm Comparisons and Ablation Studies,” explains the reporting protocol and identifies these table correspondences. FedCADC achieves the lowest standard deviation in 14 of the 18 settings in Table 13. The additional runs provide a broader empirical basis for assessing performance and variability. **Changes and location.** Section 6.2.5 and Figures 3–4 present the geometric-alignment experiments, linked to Theorem 1 through the variance identity in Eq. (102). Tables 1 and 3 report the updated mean accuracies; Appendix E.4 and Tables 13–14 provide the corresponding standard deviations. The relevant additions and revisions are highlighted in blue in the revised manuscript.
AuthorsOpenReview ↗

Response to Question 2 (Reviewer oUVs)

We thank the reviewer for requesting evaluation beyond vision and a clearer account of the computational and communication overhead. We have added two non-visual benchmarks in Appendix E.8, “Evaluation on Non-Visual Benchmarks,” together with component-level runtime measurements and communication-cost comparisons. Also following Reviewer eHEQ's request to assess computational and communication efficiency, we have expanded the corresponding analyses for the main CIFAR-10 and CIFAR-100 experiments in Appendices E.1 and E.2.

Extended evaluation beyond vision. We evaluate Yahoo Answers topic classification using TextCNN and HuffPost news classification using pretrained BERT-Tiny. Yahoo Answers uses a shared class-balanced subset of 400,000 training examples and the complete 60,000-example test set. HuffPost uses 167,221 training and 41,808 test examples across 42 categories. Both benchmarks use 100 clients, ten participants per round, three Dirichlet settings, 200 communication rounds, five local epochs, and five independent runs. Table 17 shows that FedCADC achieves the highest mean accuracy in five of the six settings, outperforming the strongest baseline in each of these five settings by 0.47–2.35 percentage points. The only exception is HuffPost at αDir=0.5\alpha_{\mathrm{Dir}}=0.5, where FedCADC achieves 64.56%, only 0.37 percentage points below the best-performing baseline, FedScar (64.93%). These results extend the evidence for FedCADC's competitive performance to text classification.

Hardware and component-level computation overhead. All runtime measurements use the same Windows workstation with an NVIDIA GeForce RTX 5090 D v2 GPU, a 24-core CPU at a base frequency of 3.70 GHz, 64 GB RAM at 5600 MT/s, and NVMe SSD storage. We report computation time summed over participating clients and the server, excluding evaluation and flatness analysis; communication is reported separately as transmission volume. Tables 8–9 give the theoretical client and server workloads, and Tables 10–11 report measured times for the main experiments. FedCADC requires two forward and two backward passes per local step, matching the propagation workload of FedSAM, with additional O(dpar)\mathcal{O}(d_{\mathrm{par}}) vector operations, where dpard_{\mathrm{par}} denotes the number of model parameters.

Table 18 reports the average training time per communication round on Yahoo Answers and HuffPost, including ablations that quantify the overhead of the curvature-aware and direction-consistency terms. Removing the curvature term eliminates the perturbation probe; restoring it approximately doubles the average runtime through the additional gradient evaluation. Adding direction consistency to FedCADC w/o RdirR_{\mathrm{dir}} increases the average time per round by only 0.723–0.886 seconds, or 2.78%–3.16%, because it requires vector operations without additional neural-network forward or backward passes. The complete method incurs 5.78%–6.07% higher average time per round than FedSAM, consistent with the approximately 6% average client-side overhead observed in the main experiments.

Communication and computational cost analysis. The curvature-aware term is computed locally and incurs no additional communication cost. Direction consistency adds one model-sized download per participating client from the second round onward. Accordingly, Table 5 counts three transmissions per round for FedCADC versus two for FedAvg and FedSAM. Accounting for the first round, its cumulative volume is mpart(3Rhit−1)Smodelm_{\mathrm{part}}(3R_{\mathrm{hit}}-1)S_{\mathrm{model}}. The non-visual model sizes are 16.16 MB for TextCNN and 17.57 MB for BERT-Tiny.

Despite these per-round overheads, FedCADC reaches the targets with the shortest computation time in 14/18 main-experiment settings (Table 12) and the lowest communication volume in 16/18 settings (Table 7). In the 13 time-winning settings where a baseline also reaches the target, its time savings over the fastest successful baseline range from 1.6% to 68.9%. On the non-visual benchmarks, Tables 19–20 show that FedCADC achieves the shortest computation time in 15 of the 18 settings and the lowest communication volume in 17 of the 18 settings. In the 15 settings with the shortest computation time, FedCADC reduces computation time by 14.0%–29.0% relative to the fastest baseline that reaches the same target. In the 17 settings with the lowest communication volume, it reduces communication by 26.1%–49.1% relative to the most communication-efficient baseline that reaches the same target. These results demonstrate FedCADC's overall advantages in communication and computational costs across the evaluated settings.

Changes and location: Appendix E.1, Table 7; Appendix E.2, Tables 8–12; and Appendix E.8, Tables 17–20. The corresponding additions and revisions are highlighted in blue in the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.