acceptodds
Under review as a conference paper at ICLR 2027

FedESO: Exploration-to-Stabilization Optimization for Model-Heterogeneous Federated Learning

Abstract

Model-heterogeneous federated learning (MH-FL) enables resource-constrained clients to train and deploy submodels of different capacities extracted from a shared global model. Achieving strong submodel performance requires both identifying promising parameter subsets and sufficiently optimizing their parameters. However, existing MH-FL methods either provide limited exploration, risking suboptimal selections, or continually vary selected submodels without explicitly regulating their stabilization, hindering consistent optimization. To bridge this gap, we propose FedESO, an exploration-to-stabilization framework that promotes early exploration through Gumbel-perturbed learned submodel selection and progressively anneals the perturbations to stabilize promising submodels for consistent optimization. We formulate dynamic MH-FL as a time-varying optimization problem and establish a nonconvex convergence guarantee governed by the cumulative variation of the induced submodel objectives. We show that sublinear cumulative objective variation is sufficient for asymptotic stationarity and, under finite-time mask stabilization, further establish stationary convergence for the resulting fixed aggregate submodel objective. Our analysis directly characterizes the resource-compatible client submodels relevant to cross-device deployment, rather than relying solely on convergence of the full-model objective. Extensive experiments across vision and natural language processing tasks under diverse resource constraints show that FedESO consistently outperforms state-of-the-art approaches, with gains of at least 10 and 2 percentage points in training-from-scratch and fine-tuning settings, respectively, while also improving optimization efficiency.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
model-heterogeneous federated learning, exploration, stabilization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_hBFYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Summary

FedESO addresses model-heterogeneous federated learning (MH-FL), where clients have different computational capacities and train only resource-compatible submodels of a shared model. Its central challenge is that components excluded from a selected submodel receive no task-dependent importance feedback, limiting exploration and potentially preventing effective submodel selection. FedESO introduces a dynamic optimization framework that learns component importance from first-order loss sensitivity, uses TopK selection with a straight-through estimator (STE) to extract structured submodels, and periodically evaluates all candidate components through scheduled full importance feedback. The feedback intervals increase over training to reduce overhead while allowing the selected submodels to become more stable.

Contributions

  1. FedESO learns importance scores from first-order loss sensitivity to guide structured submodel selection, rather than relying solely on indirect parameter-magnitude criteria.
  2. Periodic evaluation provides feedback for components excluded from the currently selected submodel, balancing exploration and computational cost.
  3. The method is evaluated on CIFAR-100 with ViT-Small and Banking-77 with BERT-Small, against HeteroFL, FedRolex, and FIARSE. The experiments are comprehensive.

Validation

Across the two benchmarks, FedESO achieves average accuracies of 39.09% on CIFAR-100 and 84.95% on Banking-77, outperforming the strongest reported baseline by 9.63 and 2.92 percentage points, respectively. An ablation on CIFAR-100 shows that scheduled full feedback improves accuracy from 35.94% without full feedback to 39.09% with feedback every 16 rounds, and more frequent feedback achieves higher accuracy but incurs greater computational cost.

Critical strengths
  1. Components excluded from the selected submodel receive no task-dependent importance feedback, which limits exploration and can lead to suboptimal submodel selection. FedESO addresses this issue through scheduled full importance feedback, providing a principled balance between exploring previously excluded components and controlling computational overhead. This directly targets a central challenge in heterogeneous federated optimization.
  2. FedESO outperforms the strongest reported baseline by 9.63 percentage points on CIFAR-100 and 2.92 percentage points on Banking-77. The feedback ablation further supports the effectiveness of periodically evaluating all candidate components, while illustrating the trade-off between accuracy and computational cost. In addition, the convergence analysis explicitly considers the time-varying objective induced by dynamic submodel selection, providing theoretical justification for studying this optimization setting. Together, these results support the method's potential value beyond a purely heuristic improvement.
Critical weaknesses
  1. The convergence analysis relies on assumptions about the cumulative variation of the time-varying objective, with sublinear variation sufficient for asymptotic stationarity. However, the paper does not establish that the proposed scheduled full importance feedback guarantees this condition or that submodel selection eventually stabilizes. Consequently, the theoretical results provide conditional convergence guarantees rather than a direct justification of the convergence behavior of the proposed algorithm.
  2. Although the paper motivates FedESO through heterogeneous client resource limitations, its experimental setup assumes that clients can store the full model and restricts the size of the submodel trained. This leaves an important practical scenario unverified, which is clients whose memory capacity is insufficient to accommodate the full model. Moreover, the evaluation covers only two benchmarks with 12 fully participating clients, limiting the evidence for generalizability to larger and more diverse federated environments. These limitations weaken the empirical support for the paper's broader claims about practical applicability.
Questions for authors
  1. Can the authors clarify whether the proposed scheduled full importance feedback theoretically guarantees sublinear cumulative objective variation, or whether convergence depends on an additional assumption that is not guaranteed by the algorithm?
  2. Can the authors clarify how FedESO can be implemented when a client's memory capacity is insufficient to store the full model, given that the experiments assume full-model storage despite restricting the size of the trained submodel?

Suggestions for future work

Evaluating FedESO with larger client populations, partial client participation, and more diverse resource constraints are helpful to establish the scalability and practical generalizability of the approach.

Reviewer_SY4TOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies model hetrogeneous federated learning, where the weak clients train and use a smllaer submodels from the global models. The authors propose to learn the importance score for the structured components using amovement pruning style gradient signal and extract submodels by TopK and found that components outside a client’s submodel never get a score update. To fix this the authors proposed to periodically send the full model to clients for score evaluation only, with the interval growing over training. Experiments on the two datasets show clear gains over the three baselines especially in the from-scratch setting.

Critical strengths

The gain from the learned section of the global model is valid even without the full-feedback setting: the variant with no full feedback beats all baselines on CIFAR-100 at the same cost as HeteroFL. The framing is also clear here, following previous work on submodel selection and optimizing the submodels that actually get deployed.

Critical weaknesses

The full-feedback step seems to undercut the premise of the paper, where the setting assumes weak clients can’t train the full model, yet in feedback rounds every client would require the forward/backward passes on the full model, which needs to be better explained. If the local clients are able to do this occationally then the whole motivation does not stand.

The theory parts are thin: Note that Theorem 1 holds for any mask sequence, and the bound is tightest when masks never change, which then operates exactly HeteroFL. The theory parts did not explain the optimality or connect prediction or interpretability to the method as well. The bounded-variation assumption is also too strong; it is close to standard local-SGD analysis with an extra term rather than the analysis of the proposed method.

Questions for authors
  • Is TopK applied globally across layers or per layer?

  • In Table 1, FedESO’s 1/64 submodel beats its 1/4 submodel even though they are nested. Is this because each tier is evaluated on different clients’ local test sets?

Reviewer_QAgxOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies FL under system heterogeneity with clients having different compute budgets and train submodels of a shared model. FedESO uses task-dependant first-order importance signals to select top-scoring components for each client to train. Further, they propose full feedback rounds that occasionally scores all components. The authors provide convergence results as well as empirical results on different settings.

Critical strengths
  1. The results are convincing specially beating heteroFL with no feedback is convincing.
  2. Table 2 ablation is informative ad intuitive tradeoff provided cleanly.
Critical weaknesses

W1. A key limitation of this work is in the full-feedback rounds. All the clients participating in a round needs to run full forward and backward pass through the entire model even the challenged clients (1/64). This becomes more problematic when majority of participating clients are constrained severely.

W2. FedESO uses about 30% more communication compared to the HeteroFL in Table 1.

W3. The convergence results is generic and holds for any method with changing masks including baselines. Is there anything specific to FedESO the proof intends to portray?

W4. There is no related works section. Can you compare against dynamic sparse training methods that use periodic exchanges e.g., RigL [1] and FedDST [2]?

[1] Evci, Utku, et al. "Rigging the lottery: Making all tickets winners." International conference on machine learning. PMLR, 2020.

[2] Bibikar, Sameer, et al. "Federated dynamic sparse training: Computing less, communicating less, yet learning better." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 36. No. 6. 2022.

Questions for authors

Q1. Is there any perturbation added to the component raking for ViT ? Q2. What is the peak training memory per tier in full feedback rounds? Q3. How many seeds were used in Table 1. and is the the final accuracy or the best accuracy?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.