acceptodds
Under review as a conference paper at ICLR 2027

FalOTS: Orthogonal Target-Aware SAM for Federated Active Learning Under Global Class Imbalance

Abstract

Federated active learning (FAL) reduces annotation costs by selectively labeling data distributed across clients. However, under global class imbalance, the class proportions in the aggregate labeled data may differ substantially from those desired for evaluation (target prior), limiting the global model's performance. Existing FAL methods mainly improve sample acquisition, leaving this mismatch insufficiently addressed during local training. Sharpness-aware minimization (SAM) offers a promising foundation for local training by seeking parameters whose loss remains low under nearby perturbations. Yet, its empirical-gradient perturbation does not explicitly account for the mismatch between global labeled and target class priors. To address this limitation, we propose , a target-aware SAM optimizer for local training within FAL loop. At each acquisition phase, constructs class weights from the ratio between the target prior and current global labeled prior. It computes a target-aware gradient on each labeled mini-batch and adds its normalized orthogonal residual to SAM perturbation. This correction preserves SAM empirical-gradient component while adding a complementary direction. The subsequent descent step uses the original unweighted loss. Our analysis characterizes the correction's geometric properties and provides sufficient conditions for a lower target risk after one update than SAM. improves global model accuracy under limited labeling budgets, outperforming FAL baselines using SGD- and SAM-based optimizers across diverse benchmarks and settings.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Federated active learning, Global class imbalance

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_EYHxOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this work, the authors propose FalOTS, a federated active learning framework that employs orthogonal, target-aware Sharpness-Aware Minimization (SAM) for optimization. During each sample acquisition phase, FalOTS constructs class weights based on the ratio between target priors and the distribution of labels already acquired globally. It computes target-aware gradients for each labeled mini-batch and incorporates the normalized orthogonal residual of these gradients into the SAM perturbation. By integrating global class information into local optimization, FalOTS achieves its objectives without requiring any modifications to the existing sample acquisition rules or server-side aggregation algorithms used in federated active learning. Extensive experiments are conducted to evaluate the performance of the proposed mechanism.

Critical strengths
  1. This paper distinguishes between heterogeneity among clients and class imbalance in the aggregate data across all clients, demonstrating that the latter can still persist. This helps clarify the distinction between existing federated SAM methods and the specific problem addressed in this paper.

  2. The presentation of the proposed method is clear and the paper is overall well-written.

  3. The evaluation of the proposed method is solid and comprehensive.

Critical weaknesses
  1. Global prior information may become completely ineffective after local projection. The entire FalOTS framework could thus degenerate into FedSAM.

  2. The method discards all parallel components, even when the target gradient points in the opposite direction to the empirical gradient.

  3. In Table 1, the Target entry for Focal-SAM is marked with a ×. However, the original Focal-SAM paper explicitly establishes a framework characterized by an imbalanced training distribution, a uniform target distribution during testing, and consistent class-conditional distributions, with the objective of mitigating the risks associated with a uniform test distribution. Therefore, this "×" at most indicates that Focal-SAM does not accept an arbitrary, user-specified πtgt\pi_{\mathrm{tgt}} as an explicitly configurable input; it cannot be interpreted to mean that the method fails to account for the discrepancy between the training prior and the target prior.

  4. In Table 2, multiple medical baselines (Random, KAFAL, LoGo) are evaluated under two heterogeneity settings: α=0.1\alpha=0.1 and α=100\alpha=100.

  5. The absence of category-specific results makes it impossible to directly demonstrate that the gains stem from addressing class imbalance.

Questions for authors

See in weaknesses.

9 comments
AuthorsOpenReview ↗

Official Author Responses to Reviewer EYHx [1/7]

We thank Reviewer EYHx for the careful review and for appreciating the distinction between client heterogeneity and global class imbalance, the clarity of the presentation, and the solidity of the evaluation. Each point is addressed below. We conducted requested experiments/discussion on (1) Potentially FalOTS degenerates into FedSAM, (2) Concern about discarding all parallel components, (3) Clarification "Target" entry for Focal-SAM, (4) Duplication number and (5) Category-specific results. The corresponding revisions are marked in blue in the revised PDF, Tab. R-numbers refer to the new results (Figures are added directly in the revision) in Appx. J.

AuthorsOpenReview ↗

Official Author Responses to Reviewer EYHx [2/7]

W1: Global prior information may become completely ineffective after local projection. The entire FalOTS framework could thus degenerate into FedSAM.

This is a legitimate concern, and it can be answered precisely.

When can the correction vanish? On a client's mini-batch, gemp=∑yπB(y)μ^yg_{\mathrm{emp}}=\sum_y\pi_{\mathcal B}(y)\hat\mu_y and gtgt∝∑yw(y)πB(y)μ^yg_{\mathrm{tgt}}\propto\sum_y w(y)\pi_{\mathcal B}(y)\hat\mu_y are two positive combinations of the same per-class mean gradients μ^y\hat\mu_y (πB\pi_{\mathcal B}: batch label frequencies). They are collinear, so that v=0v=0, only if (a) ww is constant over the classes present in the batch, i.e., the global labeled prior already matches the target on those classes, in which case reducing to FedSAM is the intended behavior (Prop. 3(2)); or (b) the per-class gradients are themselves collinear, a measure-zero event in RP\mathbb R^P. The projection removes a single direction,span{g^}\mathrm{span}\lbrace\hat g\rbrace, out of PP, and the global information, which enters through ww, lives in the remaining P−1P-1 dimensions. The weights are global by design: with a common ww, ∑kqk Ek[w∇ℓ]=∇Ltgt\sum_k q_k\,\mathbb E_k[w\nabla\ell]=\nabla L_{\mathrm{tgt}} (Prop. 1), so even a client that holds only a few classes up-weights exactly the globally rare classes among them.

Measured during training. We recorded the residual on every client's mini-batches along full FAL runs (γ=20\gamma=20, α=0.1\alpha=0.1; 7,920 probes per dataset; Tab. 17 and Fig. 14c of the revised PDF), which is exactly the local projection the reviewer asks about:

Tab. R1: Relative size of the orthogonal residual over all probed local steps. ∥v∥/∥gtgt∥\Vert v\Vert/\Vert g_{\mathrm{tgt}}\Vert is the sine of the angle between gtgtg_{\mathrm{tgt}} and g^\hat g; a value of 0 means the step reduces to FedSAM. The zero-guard of Alg. 1 sets u=0u=0 when v=0v=0.

median ∥v∥/∥gtgt∥\Vert v\Vert /\Vert g_{\mathrm{tgt}}\Vert10th pct.min.zero-guard triggered
CIFAR-100.560.120.0110%
CIFAR-1000.530.400.2480%

The orthogonal part carries about half of the target-weighted gradient's norm in a typical step and is stable across all nine phases (median 0.51–0.61).

Accuracy. Consistently, FalOTS matches or improves over FedSAM in every setting of Tabs. 5–7 and 12 (α\alpha from 0.1 to 100); the gain is largest at γ∈{20,50}\gamma\in\{20,50\} and smaller at γ=1\gamma=1 (Fig. 6b), which is what the analysis predicts as the priors approach each other. We added this argument to Sec. 3.3 and the measurements to Appx. J.3.

AuthorsOpenReview ↗

Official Author Responses to Reviewer EYHx [3/7]

W2: The method discards all parallel components, even when the target gradient points in the opposite direction to the empirical gradient.

Why the parallel component is discarded. The budget along g^\hat g is SAM's own ascent step, which is what makes the method sharpness-aware. Keeping ⟨gtgt,g^⟩g^\langle g_{\mathrm{tgt}},\hat g\rangle\hat g changes SAM's linearized increase ρ∥∇L∥\rho\|\nabla L\| (Prop. 3(1)): a positive component only rescales ρ\rho, which Fig. 6c shows is not where the gain comes from, and a negative component would partly cancel the ascent and remove sharpness control exactly when head- and tail-class gradients disagree. The sign is not lost: it enters ∥v∥2=∥gtgt∥2−⟨gtgt,g^⟩2\|v\|^2=\|g_{\mathrm{tgt}}\|^2-\langle g_{\mathrm{tgt}},\hat g\rangle^2, and under full anti-alignment the residual vanishes and FalOTS conservatively falls back to SAM. If the target gradient opposed the training gradient as a whole, we believe the remedy would belong in the descent objective rather than in the neighborhood probe.

Does the opposing case occur in practice? We measured it on all clients, rounds and phases (Appx. J.3): ⟨gtgt,g^⟩<0\langle g_{\mathrm{tgt}},\hat g\rangle<0 occurred in 0 of 15,840 probes on CIFAR-10 and CIFAR-100 as presented in Tab. R2; the median cos⁡(gtgt,gemp)\cos(g_{\mathrm{tgt}},g_{\mathrm{emp}}) is 0.83–0.85. This is expected, since both are positive combinations of the same per-class gradients. Hence a "keep only when negative" variant coincides with FalOTS at every probed step, while "always keep" (Fig. 6a, "no ortho") is consistently worse. This discussion is now in Sec. 3.3.

Tab. R2: Alignment of the target-weighted and empirical gradients along one full FAL run per dataset (γ=20\gamma=20, α=0.1\alpha=0.1; 7,920 probed local steps per run; Appx. J.3 of the revised PDF).

CIFAR-10CIFAR-100
⟨gtgt,g^⟩<0\langle g_{\mathrm{tgt}},\hat g\rangle<0, FalOTS iterates0 of 7,9200 of 7,920
⟨gtgt,g^⟩<0\langle g_{\mathrm{tgt}},\hat g\rangle<0, FedSAM iterates0 of 7,9200 of 7,920
median cos⁡(gtgt,gemp)\cos(g_{\mathrm{tgt}},g_{\mathrm{emp}}) (range over the 9 phases)0.83 (0.80–0.85)0.85 (0.79–0.86)
Reviewer_gKRPOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3(+1)
Presentation
2
Soundness
3(+1)
Read review
Paper summary

Existing FAL methods face challenges arising from distribution mismatch under global class imbalance, and existing FL-SAM methods cannot efficiently address this issue. To address this challenge, the paper proposes a target-aware SAM optimizer, FalOTS. In particular, by adding a normalized orthogonal residual to the SAM perturbation, FalOTS corrects the updated direction. Empirically, this methods outperforms baselines across different datasets.

Critical strengths
  1. The paper studies an interesting problem in FAL, where the global class is imbalanced. It identifies the limitations of existing methods.
  2. The proposed method is simple and doesn’t require a complicated design. It introduces a normalized orthogonal residual into the local step.
  3. The experimental evaluation is thorough, covering different datasets, ablation studies, and efficiency tests.
Critical weaknesses
  1. The analysis states the only method-specific input required is ∥ϵ∥≤2ρ\|\epsilon\| \leq \sqrt{2}\rho. However, it is not clear what theoretical benefits the proposed method provides. In other words, what theoretical advantages does the orthogonal target-aware correction offer over other SAM methods?
  2. To my understanding, Proposition 4 requires ⟨Hr,u⟩>0\langle Hr, u\rangle > 0 to achieve an improvement. However, the empirical study doesn’t examine whether this condition holds in practice. One additional empirical evidence would help support this claim.
  3. What does the global class imbalance γ\gamma mean? Please provide a clear explanation in the experiments section.
  4. For federated learning under global class imbalance, [1] and [2] are also related works. Does the distribution mismatch also appear in their methods, or does it only occur in SAM-based methods? Please provide further discussion.

[1] L. Wang, S. Xu, X. Wang, and Q. Zhu. Addressing class imbalance in federated learning, AAAI 2021. [2] X. Zhang, Y. Zhang, T. Yang, R. Souvenir, and H. Gao. Federated compositional deep auc maximization, NeurIPS 2023.

Questions for authors

See the Weaknesses above.

8 comments
AuthorsOpenReview ↗

Official Author Responses to Reviewer gKRP [1/6]

We thank Reviewer gKRP for the constructive review and for finding the problem interesting, the method simple, and the evaluation thorough. Each point is addressed below. We conducted requested experiments/discussion on (1) Actual theoretical advantage of the orthogonal correction over SAM methods, (2) Empirical check of the curvature condition, (3) Clarifying about global class-imbalance ratio, and (4) Discussion about additional related works. The corresponding revisions are marked in blue in the revised PDF, Tab. R-numbers refer to the new results (Figures are added directly in the revision) in Appendix J.

AuthorsOpenReview ↗

Official Author Responses to Reviewer gKRP [2/6]

W1: The analysis states the only method-specific input required is |ε| ≤ √2ρ. However, it is not clear what theoretical benefits the proposed method provides. In other words, what theoretical advantages does the orthogonal target-aware correction offer over other SAM methods?

We clarify that the quoted sentence ("the only method-specific input required is ∥ϵ∥≤ρ2\|\epsilon\|\le\rho\sqrt2") comes from the within-phase convergence analysis (Appx. D). That analysis is deliberately method-agnostic: it shows that adding the correction does not break FedSAM-type stationarity (same O(R−1/2)O(R^{-1/2}) rate, with the perturbation entering only through Λ2ρ2\Lambda^2\rho^2 terms). We should have stated more clearly that this is a no-harm result and that the advantage is established in Secs. 3–4. We now make this explicit and add a proposition. Relative to other SAM variants, the orthogonal correction provides:

  1. SAM's sharpness term is kept exactly (Prop. 3(1)): ⟨∇L,ϵFalOTS⟩=ρ∥∇L∥\langle\nabla L,\epsilon_{\mathrm{FalOTS}}\rangle=\rho\|\nabla L\|. Any replacement direction d≠g^d\neq\hat g (ImbSAM's tail-class gradient, a target-weighted gradient) gives ⟨∇L,ρd⟩<ρ∥∇L∥\langle\nabla L,\rho d\rangle<\rho\|\nabla L\| and loses part of SAM's first-order sharpness control.
  2. The added budget is used optimally for the target (Prop. 2): among unit directions that are first-order neutral to the training loss, uu uniquely maximizes the increase of the target-weighted loss, and no choice of ρ\rho reaches it from SAM.
  3. New Prop. 5 (dominance): ⟨gtgt,ϵFalOTS⟩=⟨gtgt,ϵSAM⟩+ρ∥v∥\langle g_{\mathrm{tgt}},\epsilon_{\mathrm{FalOTS}}\rangle=\langle g_{\mathrm{tgt}},\epsilon_{\mathrm{SAM}}\rangle+\rho\|v\| at no first-order cost to the training loss, and FalOTS is the unique perturbation ρg^+ρd\rho\hat g+\rho d, d⊥g^d\perp\hat g, attaining this. With gtgt=∇Ltgtg_{\mathrm{tgt}}=\nabla L_{\mathrm{tgt}} (Prop. 1) this is a statement about the target risk.
  4. A one-step guarantee on the target risk (Prop. 4): lower than SAM's whenever ⟨Hr,u⟩>0\langle Hr,u\rangle>0. To our knowledge, neither federated SAM methods (FedSAM, FedLESAM, FedGMT) nor class-aware ones (ImbSAM, Focal-SAM) give a statement about a specified target risk. Empirically, the condition holds in 87–89% of local steps in practice (reply to W2).

These properties have a direct empirical counterpart in the new Tab. R1: replacing SAM's ascent direction by the target-weighted gradient (W-SAM) can improve slightly over FedSAM in average but still clearly stays below FalOTS in all four CIFAR-10/100 settings. We further provide other weighting alternatives to better demonstrate the robustness of FalOTS.

Tab. R1: Comparison of FalOTS with other simple weighted alternatives. Bold is the single best method. Italic is the best combined method.

Method (FairFAL acquisition, FedAvg, γ=20\gamma=20, 45% budget)CIFAR-10 α=0.1\alpha=0.1CIFAR-10 α=100\alpha=100CIFAR-100 α=0.1\alpha=0.1CIFAR-100 α=100\alpha=100
FairFAL (SGD, unweighted)60.4664.5329.2430.03
FairFAL (SGD) + Ratio Loss60.6264.6829.4230.10
CB-SGD: SGD on the weighted loss LwL_w (w=πtgt/πempw=\pi_{\mathrm{tgt}}/\pi_{\mathrm{emp}})60.1264.2729.1629.74
CB-SGD: class-balanced sampling60.5064.8529.0328.66
FedSAM62.0466.9130.7331.18
FedSAM + Ratio Loss62.2767.1430.8831.26
FedSAM, compute-matched (T=150T=150 rounds)62.3066.7730.8131.23
W-SAM: ascent along gtgt/∥gtgt∥g_{\mathrm{tgt}}/\|g_{\mathrm{tgt}}\|, unweighted descent62.4066.6930.9431.36
W-SAM, compute-matched (T=150T=150 rounds)62.3466.7230.9831.45
CB-SAM: ascent and descent on LwL_w61.8367.1230.5930.97
FalOTS63.2868.2831.9531.63
FalOTS + Ratio Loss63.4968.4532.1731.79

Revisions. Sec. 4 now separates the two layers explicitly (no-harm in Appx. D; advantage in Props. 2–5), Prop. 5 is added with its proof in Appx. E.6. We also added new experimental results at Appx. J.1 (Tab. 15).

AuthorsOpenReview ↗

Official Author Responses to Reviewer gKRP [3/6]

W2: To my understanding, Proposition 4 requires ⟨Hr, u⟩ > 0 to achieve an improvement. However, the empirical study doesn't examine whether this condition holds in practice. One additional empirical evidence would help support this claim.

Thank you for the suggestion; we measured it directly on CIFAR-10 and CIFAR-100.

Protocol. One full FAL run per dataset (γ=20\gamma=20, α=0.1\alpha=0.1, nine phases). At rounds {1,10,…,100}\{1,10,\dots,100\} of each phase, for 10 clients and 8 fixed mini-batches each (7,920 probes per dataset), we compute uu as in Alg. 1, HuHu by Hessian-vector product, and A⊥=⟨r,Hu⟩A_\perp=\langle r,Hu\rangle, with rr estimated by the weighted gradient on the global labeled set (Prop. 1) or by an oracle on a held-out class-balanced split. From the same θ\theta and mini-batch we also measure the actual one-step difference Δ=Ltgt(θFalOTS+)−Ltgt(θSAM+)\Delta=L_{\mathrm{tgt}}(\theta_{\mathrm{FalOTS}}^{+})-L_{\mathrm{tgt}}(\theta_{\mathrm{SAM}}^{+}).

Tab. R2: Empirical check of the curvature condition of Prop. 4, measured along one full FAL run per dataset. Each cell gives the percentage over all probed rounds, and in parentheses the percentage excluding the first round of each phase, where the model has just been re-initialized. The target gradient rr is estimated either by the weighted gradient on the global labeled set (Prop. 1) or by an oracle on a held-out class-balanced split. Full results are provided in Tab. 16 (Appx. J.2) of the revised PDF.

% of probed local steps: all rounds (excl. round 1)CIFAR-10, global est.CIFAR-10, oracleCIFAR-100, global est.CIFAR-100, oracle
A⊥=⟨Hr,u⟩>0A_\perp=\langle Hr,u\rangle>080.3 (83.6)84.8 (88.6)76.1 (78.6)83.2 (87.2)
Δ<0\Delta<0 (FalOTS target loss below SAM's)79.9 (83.4)83.3 (87.1)84.3 (88.0)87.6 (91.8)
sign$(\Delta)$ = sign$(-A_\perp)$ (Eq. 15)87.8 (89.2)91.1 (93.2)79.6 (81.5)86.5 (89.3)

We have three observations as follows:

  • The condition holds in the large majority of local steps on both datasets and with both estimators, and the actual one-step target loss of FalOTS is lower than SAM's in 83–92% of steps (oracle).
  • The exception is the first round of each phase, right after re-initialization, where A⊥≈0A_\perp\approx0 and is positive in about half of the probes. From round 10 on, the fraction is 83–92% in every round (Fig. 14a) and 76–91% in every phase.
  • Eq. 15 predicts the sign of the actual update in 86–93% of steps (oracle), so it describes training at our η\eta and ρ\rho, not only the small-step limit.

Full statistics, including a local-gradient estimator, are in Tab. 16 and Fig. 14 (Appx. J.2).

Reviewer_wWeKOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This article proposes FalOTS, an optimization method for federated active learning in scenarios with global class imbalance, characterized by both "target-awareness" and "sharpness-awareness." The method distinguishes between the discrepancy in global prior class distributions and the heterogeneity among clients; to do so, it utilizes the ratio between the prior distributions of target classes and the empirical prior class distributions to construct a "target-aware gradient." Specifically, the component of this gradient—which is orthogonal to the empirical gradient (following normalization)—is incorporated into the perturbation used in the Sharpness-Aware Minimization (SAM) method, while the subsequent descent step is calculated using the unweighted empirical loss function.

Critical strengths
  1. Well-founded problem. The paper clearly distinguishes between global class imbalance and client-side heterogeneity, providing empirical evidence that active acquisition alone does not necessarily eliminate the mismatch in the prior distributions of the target classes of interest.

  2. Simple and interpretable method. The orthogonal decomposition preserves the empirical SAM (Sampling-based/Statistical Alignment Method) perturbation component while incorporating complementary information focused on the target classes. This method can be integrated into existing Federated Active Learning (FAL) workflows without requiring modifications to acquisition or model aggregation procedures.

  3. Comprehensive empirical evaluation. Experiments conducted on eight benchmark datasets for image and text classification—using various SAM-based baselines and under different heterogeneity scenarios—demonstrate generally consistent improvements. The mathematical analysis precisely distinguishes between geometric properties and guarantees for improving conditional risk with respect to the target classes.

Critical weaknesses
  1. Insufficient isolation of the main contribution. Although the paper compares the proposed method against SAM variants that account for class information and includes ablation studies on orthogonality, the evidence does not conclusively demonstrate that the proposed orthogonal correction outperforms simpler alternatives—such as SAM with weighting based on target priors or class-balanced training—under comparable computational budgets. This limits the scope of the algorithmic contribution presented.

  2. Discrepancy between theoretical motivation and practical improvement. The single-step objective risk comparison relies on a quantity related to curvature, for which a favorable sign is not guaranteed. Furthermore, normalizing an orthogonal residual—which could be arbitrarily small yet non-zero—to unit length may introduce instability when working with noisy estimates derived from mini-batches. The paper provides little empirical evidence linking these issues to actual optimization behavior.

  3. Computational efficiency is not sufficiently demonstrated. FalOTS introduces an additional gradient evaluation compared to FedSAM. While it may reduce the number of communication rounds, it remains unclear whether the accuracy advantage persists when considering a total computational budget or an equivalent end-to-end execution time.

Questions for authors
  1. Can the authors compare FalOTS with target-prior-weighted SAM and with an SGD baseline using balanced classes, while employing the same acquisition strategy and comparable computational budgets?

  2. How sensitive is FalOTS to small orthogonal residual norms? Could residual thresholding or a magnitude-aware correction improve stability, particularly when the empirical and target prior distributions are nearly identical?

  3. Can the authors quantify the end-to-end computational trade-off in comparison to FedSAM, including results obtained under equivalent computational budgets?

8 comments
AuthorsOpenReview ↗

Official Author Responses to Reviewer wWeK [1/6]

We thank Reviewer wWeK for the detailed review and for recognizing the well-founded problem, the simplicity and interpretability of the method, and the breadth of the comprehensive empirical evaluation. We address each point below. We conducted requested experiments on (1) Simpler alternative comparison, (2) Theory vs. practice: sign of the curvature term; normalising small residuals; empirical link. (3) End-to-end computational trade-off, and (4) Sensitivity to small residual norms; thresholding and magnitude-aware correction. The corresponding revisions are marked in blue in the revised PDF, Tab. R-numbers refer to the new results (Figures are added directly in the revision).

AuthorsOpenReview ↗

Official Author Responses to Reviewer wWeK [2/6]

W1: Insufficient isolation of the main contribution. Although the paper compares the proposed method against SAM variants that account for class information and includes ablation studies on orthogonality, the evidence does not conclusively demonstrate that the proposed orthogonal correction outperforms simpler alternatives—such as SAM with weighting based on target priors or class-balanced training—under comparable computational budgets. This limits the scope of the algorithmic contribution presented.

Q1: Can the authors compare FalOTS with target-prior-weighted SAM and with an SGD baseline using balanced classes, while employing the same acquisition strategy and comparable computational budgets?

We thank the reviewer for this important point. We agree that the contribution should be isolated against simpler reweighting alternatives, and we added these comparisons. We first summarise what submission already isolates, then new experiments.

What submission already isolates.

  1. Direction vs. radius. Fig. 6c re-runs every SAM baseline with the enlarged radius ρ2\rho\sqrt{2} (the total norm of our perturbation); none closes the gap, so the gain is not a larger perturbation.
  2. Orthogonal residual vs. full target gradient. Fig. 6a compares our correction with the variant that keeps the full normalised target-weighted gradient. Keeping the aligned component is clearly worse than the orthogonal residual.
  3. Where the weights act. Fig. 6d applies the target-aware correction in the ascent step, in the descent step, or in both; ascent-only is best, which is why FalOTS keeps the unweighted descent loss.
  4. Class-weighted SAM baselines. ImbSAM (tail-class ascent) and Focal-SAM (class-weighted sharpness) are included in every table.
  5. Same acquisition, no SAM. FairFAL trained with SGD is exactly the "same acquisition strategy" baseline (Sec. 5, Implementation details) and appears in Tabs. 5-7 and Fig. 4.

What was missing. The simple alternatives the reviewer suggested: SAM whose ascent direction is the target-weighted gradient (W-SAM), SAM with weighted ascent and descent (CB-SAM), class-balanced SGD on the weighted loss (CB-SGD) and the Ratio Loss [R1], under the same settings (new Tab. R1):

Tab. R1: Comparison of FalOTS with other simple weighted alternatives. Bold is the single best method across 5 seeds. The 45% summary is Tab. 3 and the full CIFAR-100 table (all budgets) is Tab. 15 (Appx. J.1) of the revised PDF.

Method (FairFAL acquisition, FedAvg, γ=20\gamma=20, 45% budget)CIFAR-10 α=0.1\alpha=0.1CIFAR-10 α=100\alpha=100CIFAR-100 α=0.1\alpha=0.1CIFAR-100 α=100\alpha=100
FairFAL (SGD, unweighted)60.4664.5329.2430.03
FairFAL (SGD) + Ratio Loss60.6264.6829.4230.10
CB-SGD: SGD on the weighted loss LwL_w (w=πtgt/πempw=\pi_{\mathrm{tgt}}/\pi_{\mathrm{emp}})60.1264.2729.1629.74
CB-SGD: class-balanced sampling60.5064.8529.0328.66
FedSAM62.0466.9130.7331.18
FedSAM + Ratio Loss62.2767.1430.8831.26
FedSAM, compute-matched (T=150T=150 rounds)62.3066.7730.8131.23
W-SAM: ascent along gtgt/∥gtgt∥g_{\mathrm{tgt}}/\|g_{\mathrm{tgt}}\|, unweighted descent62.4066.6930.9431.36
W-SAM, compute-matched (T=150T=150 rounds)62.3466.7230.9831.45
CB-SAM: ascent and descent on LwL_w61.8367.1230.5930.97
FalOTS63.2868.2831.9531.63
FalOTS + Ratio Loss63.4968.4532.1731.79

All variants use the same settings and five seeds; the only thing that changes is how the class weights w=πtgt/πempw=\pi_{\mathrm{tgt}}/\pi_{\mathrm{emp}} enter the optimizer. Overall, FalOTS is the best single method in all four settings, and the comparison isolates the orthogonal construction itself, rather than the class weights or the additional gradient evaluation, as the source of the improvement. Four observations:

  1. Weighting the descent objective does not help. Class-balanced SGD is on par with or below unweighted FairFAL, and the Ratio Loss adds at most +0.23+0.23 pp on SGD or FedSAM.
  2. Replacing SAM's ascent direction is the strongest alternative but still falls short. W-SAM (ascent along gtgtg_{\mathrm{tgt}}) slightly improve over FedSAM in average and still below FalOTS in every setting. CB-SAM is indistinguishable from FedSAM.
  3. The gain is not extra compute. FedSAM with 1.5×1.5\times rounds (the same number of gradient evaluations as FalOTS) improve over FedSAM but still lower than FalOTS performance. More comparison under same acquisition strategy and comparable computational budgets are provided in Tab. R3 (W3).
  4. Objective-level corrections are complementary. Adding the Ratio Loss [R1] on top of FalOTS gives a further gains in all four settings, supporting our point that FalOTS corrects a different channel (the perturbation direction) than loss reweighting.

We added Tab. R1 and this discussion to Sec. 5.2 (Tab. 3) and Appxs. J.1 and J.5 of the PDF revision.

[R1] Wang et al. Addressing class imbalance in federated learning. AAAI 2021.
AuthorsOpenReview ↗

Official Author Responses to Reviewer wWeK [3/6]

W2: Discrepancy between theoretical motivation and practical improvement. The single-step objective risk comparison relies on a quantity related to curvature, for which a favorable sign is not guaranteed. Furthermore, normalizing an orthogonal residual—which could be arbitrarily small yet non-zero—to unit length may introduce instability when working with noisy estimates derived from mini-batches. The paper provides little empirical evidence linking these issues to actual optimization behavior.

We appreciate this careful reading and separate the three concerns. We measured all three aspects along full FAL runs:

Protocol. One full FAL run per dataset (γ=20\gamma=20, α=0.1\alpha=0.1, nine phases). At rounds {1,10,…,100}\{1,10,\dots,100\} of each phase, for 10 clients and 8 fixed mini-batches each (7,920 probes per dataset), we compute uu as in Alg. 1, HuHu by Hessian-vector product, and A⊥=⟨r,Hu⟩A_\perp=\langle r,Hu\rangle, with rr estimated by the weighted gradient on the global labeled set (Prop. 1) or by an oracle on a held-out class-balanced split. From the same θ\theta and mini-batch we also measure the actual one-step difference Δ=Ltgt(θFalOTS+)−Ltgt(θSAM+)\Delta=L_{\mathrm{tgt}}(\theta_{\mathrm{FalOTS}}^{+})-L_{\mathrm{tgt}}(\theta_{\mathrm{SAM}}^{+}).

Tab. R2: Mechanism diagnostics measured along one full FAL run per dataset. Rows (a) test the curvature condition of Prop. 4 with the oracle target gradient (held-out class-balanced split); each cell gives the value over all probes, and in parentheses the value excluding the first round of each phase (same format as Tab. 16 of the revised PDF). Rows (b) give the size of the orthogonal residual over all probes.

Measured quantityCIFAR-10CIFAR-100
(a) ⟨Hr,u⟩>0\langle Hr,u\rangle>0 (% of local steps)84.8 (88.6)83.2 (87.2)
(a) one-step target loss of FalOTS below SAM's (%)83.3 (87.1)87.6 (91.8)
(a) sign of that difference predicted by Eq. 15 (%)91.1 (93.2)86.5 (89.3)
(b) median ∥v∥/∥gtgt∥\Vert v\Vert /\Vert g_{\mathrm{tgt}}\Vert0.560.53
(b) minimum ∥v∥/∥gtgt∥\Vert v\Vert /\Vert g_{\mathrm{tgt}}\Vert0.0110.248
(b) steps with ∥v∥/∥gtgt∥<0.05\Vert v\Vert /\Vert g_{\mathrm{tgt}}\Vert <0.05 (%)1.70.0
(b) zero-guard triggered (%)00

(a) rows: the first round of each phase follows re-initialization of the model, and there ⟨Hr,u⟩>0\langle Hr,u\rangle>0 holds in only 47% (CIFAR-10) and 43% (CIFAR-100) of probes, which lowers the all-probe values.

1) Sign of ⟨Hr,u⟩\langle Hr,u\rangle. We agree it is not guaranteed: We clarify that Eq. 15 is a conditional one-step statement (Appx. E.5 shows that no unconditional claim is possible), while Props. 2–4 are unconditional. Empirically, the table shows that the condition holds, and that FalOTS actually lowers the one-step target loss relative to SAM, in the large majority of local steps on both datasets.

2) Normalizing small residuals. Small residuals are rare, and on CIFAR-100 the residual never falls below 0.24. When one occurs, uu is still a bounded (norm ρ\rho) direction that is first-order neutral to the training loss (Eq. 10), the same normalization SAM applies to its own gradient, so it cannot destabilize the step. Thresholding variants: please see our response for Q2.

3) Link to optimization behavior. Fig. 14 (Appx. J.2 of the revised PDF) plots these quantities per round over training. From round 10 on, the condition and the target-loss advantage hold in 83–97% of steps in every round, and the residual stays near 0.5 throughout.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Official Author Global Responses to AC & All Reviewers

We thank all reviewers for their careful and constructive reviews. We are encouraged that they found the problem well-motivated and clearly distinguished from client heterogeneity (wWeK, gKRP, EYHx), the method simple and interpretable (wWeK, gKRP), the presentation clear (EYHx), and the evaluation thorough (wWeK, gKRP, EYHx).

The shared question is whether the gain comes from the orthogonal target-aware direction itself and whether the first-order analysis describes actual training. We provided new results (Appendix J of the revised PDF; changes in blue):

  1. The orthogonal construction is what helps (wWeK W1/Q1, W3/Q3; gKRP W1, W4). With the same weights, acquisition rule and radius grid, every simpler use of the target prior (target-weighted ascent W-SAM, weighted SAM, class-balanced SGD, the Ratio Loss of Wang et al., 2021) and FedSAM with equal gradient evaluations stay below FalOTS on CIFAR-10/100 under both α\alpha (Tab. 3 and Tab. 15 of the revised PDF). The strongest, W-SAM, is still clearly below FalOTS on average.
  2. The curvature condition of Prop. 4 holds in practice (wWeK W2; gKRP W2). Measured with Hessian-vector products along full FAL runs (7,920 probes per dataset; Tab. 16 and Fig. 14 of the revised PDF): after the first round of each phase, ⟨Hr,u⟩>0\langle Hr,u\rangle>0 in 88.6% (CIFAR-10) and 87.2% (CIFAR-100) of local steps, and the one-step target loss of FalOTS is below SAM's in 87.1% and 91.8%.
  3. The correction does not degenerate (EYHx W1, W2; wWeK W2, Q2). The median ∥v∥/∥gtgt∥\|v\|/\|g_{\mathrm{tgt}}\| is 0.56 / 0.53; on CIFAR-100 it never falls below 0.24; ⟨gtgt,g^⟩<0\langle g_{\mathrm{tgt}},\hat g\rangle<0 occurred in 0 of 15,840 probes (Tab. 17 and Fig. 14c of the revised PDF).
  4. The gains come from under-represented classes (EYHx W5). On CIFAR-100 at the 45% budget, FalOTS improves tail-class accuracy by +2.22 pp over FedSAM and +1.50 pp over Focal-SAM while preserving head-class accuracy (Tab. 20 and Fig. 15 of the revised PDF).
  5. Fixes. New Prop. 5 (gKRP W1), definition of γ\gamma (gKRP W3), Tab. 1 (EYHx W3), and the duplicated α=100\alpha=100 rows of Tab. 2 (EYHx W4).

We again thank the AC and reviewers for the constructive feedback, which significantly enhanced the clarity, theory and empirical support of the manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.