- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
Summary
FedESO addresses model-heterogeneous federated learning (MH-FL), where clients have different computational capacities and train only resource-compatible submodels of a shared model. Its central challenge is that components excluded from a selected submodel receive no task-dependent importance feedback, limiting exploration and potentially preventing effective submodel selection. FedESO introduces a dynamic optimization framework that learns component importance from first-order loss sensitivity, uses TopK selection with a straight-through estimator (STE) to extract structured submodels, and periodically evaluates all candidate components through scheduled full importance feedback. The feedback intervals increase over training to reduce overhead while allowing the selected submodels to become more stable.
Contributions
- FedESO learns importance scores from first-order loss sensitivity to guide structured submodel selection, rather than relying solely on indirect parameter-magnitude criteria.
- Periodic evaluation provides feedback for components excluded from the currently selected submodel, balancing exploration and computational cost.
- The method is evaluated on CIFAR-100 with ViT-Small and Banking-77 with BERT-Small, against HeteroFL, FedRolex, and FIARSE. The experiments are comprehensive.
Validation
Across the two benchmarks, FedESO achieves average accuracies of 39.09% on CIFAR-100 and 84.95% on Banking-77, outperforming the strongest reported baseline by 9.63 and 2.92 percentage points, respectively. An ablation on CIFAR-100 shows that scheduled full feedback improves accuracy from 35.94% without full feedback to 39.09% with feedback every 16 rounds, and more frequent feedback achieves higher accuracy but incurs greater computational cost.
Critical strengths
- Components excluded from the selected submodel receive no task-dependent importance feedback, which limits exploration and can lead to suboptimal submodel selection. FedESO addresses this issue through scheduled full importance feedback, providing a principled balance between exploring previously excluded components and controlling computational overhead. This directly targets a central challenge in heterogeneous federated optimization.
- FedESO outperforms the strongest reported baseline by 9.63 percentage points on CIFAR-100 and 2.92 percentage points on Banking-77. The feedback ablation further supports the effectiveness of periodically evaluating all candidate components, while illustrating the trade-off between accuracy and computational cost. In addition, the convergence analysis explicitly considers the time-varying objective induced by dynamic submodel selection, providing theoretical justification for studying this optimization setting. Together, these results support the method's potential value beyond a purely heuristic improvement.
Critical weaknesses
- The convergence analysis relies on assumptions about the cumulative variation of the time-varying objective, with sublinear variation sufficient for asymptotic stationarity. However, the paper does not establish that the proposed scheduled full importance feedback guarantees this condition or that submodel selection eventually stabilizes. Consequently, the theoretical results provide conditional convergence guarantees rather than a direct justification of the convergence behavior of the proposed algorithm.
- Although the paper motivates FedESO through heterogeneous client resource limitations, its experimental setup assumes that clients can store the full model and restricts the size of the submodel trained. This leaves an important practical scenario unverified, which is clients whose memory capacity is insufficient to accommodate the full model. Moreover, the evaluation covers only two benchmarks with 12 fully participating clients, limiting the evidence for generalizability to larger and more diverse federated environments. These limitations weaken the empirical support for the paper's broader claims about practical applicability.
Questions for authors
- Can the authors clarify whether the proposed scheduled full importance feedback theoretically guarantees sublinear cumulative objective variation, or whether convergence depends on an additional assumption that is not guaranteed by the algorithm?
- Can the authors clarify how FedESO can be implemented when a client's memory capacity is insufficient to store the full model, given that the experiments assume full-model storage despite restricting the size of the trained submodel?
Suggestions for future work
Evaluating FedESO with larger client populations, partial client participation, and more diverse resource constraints are helpful to establish the scalability and practical generalizability of the approach.