- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces HO-FL, a hybrid-order FL framework that trains a model’s bottom segment with ZO optimization and its top segment with FO optimization. Each device can flexibly select its order boundary according to its memory budget while participating in the training of the same global model. Moreover, their convergence analysis reveals a new, fundamental trade-off: clients with larger FO-trained segments can provide more accurate updates, but favoring them can underrepresent other clients’ data. The authors connect this trade-off to the bias and variance of actual multi-step local updates, yielding a sampling optimization problem and a practical dimension-aware approximation with direct model averaging. Experiments on language tasks examine task performance, client memory, and sampling under data heterogeneity. The results show that hybrid-order local training can retain the full-FO performance with lower client memory requirements.
Critical strengths
S1. The problem is critical for deploying large models on resource-constrained edge devices.
S2. The proposed framework demonstrates promising memory savings.
Critical weaknesses
W1. The paper should fucus on generalizing the results to edge federated learning beyond the evaluated configurations, e.g., by including more realistic hardware heterogeneity, additional workloads and adaptation settings, stronger non-IID scenarios, and varying client participation.
W2. The evaluation should clarify how memory budgets, order boundaries, data distributions, and client availability are assigned. It would be useful to evaluate the cases where resource availability and data distributions are correlated.
Questions for authors
Q1. Under what conditions does the dimension-aware sampling policy improve convergence to reach the uniform-client objective, considering the bias introduced by nonuniform sampling?
Q2. How closely does the proposed dimension-based score approximate the ideal sampling solution in Eqs.13-16?
Q3. Why does increasing \beta degrade performance under severe non-IID data, and can the policy detect when uniform sampling would be preferable?
Q4. How much of the reported benefit comes from heterogeneous order boundaries, as opposed to hybrid-order optimization or client sampling?
Q5. How sensitive are the results to resetting AdamW states at every communication round?
Q6. Do the reported memory savings translate into lower end-to-end training time and energy consumption on real-world heterogeneous edge devices?
5 comments
Response to Reviewer eAqG (Part 2/2)
Q4. How much of the reported benefit comes from heterogeneous order boundaries, as opposed to hybrid-order optimization or client sampling?
To separate the gains of thess components, we compare four configurations:
| Method | OPT / SST-2 IID | Qwen / AG News () | Qwen / SQuAD F1 |
|---|---|---|---|
| DeComFL (Pure-ZO) | 51.95 ± 0.00 | 55.83 ± 0.08 | 73.14 ± 0.16 |
| HO-FL (Uniform highest ZO boundary, ) | 79.47 ± 0.11 | 63.22 ± 1.44 | 75.95 ± 0.05 |
| HO-FL (Heterogeneous, ) | 86.01 ± 0.11 | 85.51 ± 0.43 | 85.34 ± 0.03 |
| HO-FL (Heterogeneous, ) | 86.54 ± 0.24 | 86.69 ± 0.56 | 86.92 ± 0.10 |
- Hybrid-order vs. pure-ZO (Row 1 vs. Row 2): Hybrid-order training provides a substantial accuracy jump over pure-ZO even under the uniform highest ZO boundary.
- Utilizing heterogeneous resources (Row 2 vs. Row 3): Allowing each client to use its available memory instead of forcing all clients to the uniform highest ZO boundary yields large gains on all tasks.
- Dimension-aware sampling (Row 3 vs. Row 4): Adding provides further consistent improvements across all three tasks.
Q5. How sensitive are the results to resetting AdamW states at every communication round?
We evaluated resetting versus persisting local AdamW states on OPT / SST-2 with HO-FL:
| Local AdamW State | Accuracy (%) | |
|---|---|---|
| 0.0 | Reset per round (Default) | 86.01 ± 0.11 |
| 0.0 | Persistent local states | 85.67 ± 0.18 |
| 0.75 | Reset per round (Default) | 86.54 ± 0.24 |
| 0.75 | Persistent local states | 86.32 ± 0.30 |
On OPT/SST-2 IID, persisting local states shows no gain over resetting, while requiring extra per-client storage and state management for server replay.
W1 & Q6. Generalization beyond evaluated configurations and real-world end-to-end time/energy.
We agree that broader system tests would strengthen our empirical scope. However, this paper mainly focuses on designing HO-FL with convergence guarantees for memory-heterogeneous FL. Within this scope, achieving near-full-FO accuracy under heterogeneous memory limits is still a highly appealing property for edge scenarios. To demonstrate this feasibility, we measured peak memory, uplink payload, and per-step local compute time on Qwen2.5-1.5B () across RTX 4090 24 GB, RTX 5090 32 GB, and A100 80 GB:
| Method | Peak Mem (GiB) on A100 | Uplink (MB/client/round) | 4090 (ms/step) | 5090 (ms/step) | A100 (ms/step) |
|---|---|---|---|---|---|
| FedAvg | 40.261 | 4.358 | OOM | OOM | 733.5 |
| HO cut8 | 29.961 | 3.113 | OOM | 983.6 | 794.2 |
| HO cut16 | 19.521 | 1.868 | 1291.4 | 1034.8 | 852.9 |
| HO cut24 | 9.080 | 0.623 | 1356.4 | 1100.4 | 913.2 |
| DeComFL | 3.917 | 0.000058 | 1177.2 | 821.4 | 1566.2 |
Since our experiments run serially on a single GPU rather than across parallel physical devices and real networks, end-to-end training time is not meaningful here. However, the above results show that FedAvg goes OOM on both 4090 and 5090, whereas HO-FL runs within each GPU's memory limit at comparable per-step latency. We added real-world edge deployment, time/energy profiling, and varying participation to Limitations and Future Work.
W2. The evaluation should clarify how memory budgets, order boundaries, data distributions, and client availability are assigned. It would be useful to evaluate the cases where resource availability and data distributions are correlated.
We have clarified our settings in Section 6.1 of the revision:
- Memory budgets: The clients are evenly partitioned into three fixed tiers of 10 clients each (Table 1), corresponding to per-device memory limits (e.g., 12/24/32 GiB on Qwen2.5-1.5B at ).
- Data distributions: Benchmarks use IID partitions, while AG News additionally evaluates label skew () with 4,000 samples per client.
- Client availability & sampling: All 30 clients are modeled as always available, with clients sampled per round.
We agree resource–data correlation is a valuable future direction. Theoretically, Theorem 4.1 and Theorem C.9 already hold without assuming independence between resource and data distributions, because the bound holds for arbitrary resource–data assignments. Thus, our framework extends naturally to correlated settings by selecting a smaller (as discussed in Appendix D.2) or incorporating tier-stratified sampling to control representation bias.
Gentle Reminder and Brief Summary
Dear Reviewer eAqG,
Thank you again for your time and thoughtful feedback. With the discussion deadline approaching, we wanted to share a quick recap of our responses:
- Theory & -Selection (Q1–Q3): Added explicit bound-improvement conditions in Section 5, clarified Eqs. (13–16) as a qualitative proxy, and added a low-cost -selection pilot within 0.11–0.68 pp of the best fixed .
- Ablations & AdamW State (Q4–Q5): Isolated the gains of hybrid-order updates, heterogeneous boundaries, and sampling, and confirmed that resetting AdamW states per round outperforms persistent states without extra state management.
- Cross-GPU Feasibility & Setup (W1, W2, Q6): Added memory, uplink, and step-time benchmarks across RTX 4090, RTX 5090, and A100, clarified our setup in Section 6.1, and noted edge deployment and resource–data correlation in Section 7.
We would be grateful to know whether these updates address your concerns. Thank you again for your consideration!
Best regards,
The Authors
Response to Reviewer eAqG (Part 1/2)
We are grateful for the time you have taken to review our paper and for your insightful review. Below we address Q1–Q3 first, followed by Q4–Q6 and W1–W2 in our second reply.
To identify this condition, we use Theorem C.9 in Appendix C.5. Substituting our sampling policy pi=(1−β)N1+βri into Eq. 77 gives: T1∑t=0T−1E∥∇f(wt)∥2≤TCrate1+q(1−β)Du+βDR+6ζ2β2(KN−1), where Du=N1∑idiZO and DR=K1∑i∈RdiZO.
Compared against uniform sampling at β=0, non-uniform sampling yields a tighter convergence upper bound when: TCrate1+q(1−β)Du+βDR+6ζ2β2(KN−1)<TCrate1+qDu. We have added this discussion to Section 5 of our revision.
Our dimension-based policy is a qualitative surrogate without a formal approximation-gap guarantee to Eq. (13). It upper-bounds the directional bias via Eq. (14), replaces dynamic variance-and-drift costs with static ZO dimensions, and restricts p to the linear mixture in Eq. (16). This preserves the trade-off between lower ZO noise and uniform participation when ZO noise dominates over client drift, and our β-sweeps in Table 4 confirm that this zero-overhead policy works well in practice.
Setting a higher β concentrates sampling on a small subset of clients with larger FO segments; under severe non-IID skew, this subset no longer represents the global data distribution and biases the aggregated model. Formally, Theorem 4.1 gives the bound O((1+q1∑ipidiZO)/T)+O(ζ2β2(N/K−1)), where the first term captures ZO gradient variance and the second captures sampling representation bias. When data heterogeneity ζ2 is large, the ζ2β2 bias term dominates, so increasing β can hurt performance.
Our policy does not detect sampling bias online during training. Instead, Appendix D.2 provides pre-training selection guidelines, including a short pilot validation across β∈{0,0.25,0.5,0.75}. To demonstrate its effectiveness, we conducted a pilot experiment on Qwen / AG News using 150 local steps and 2,400 samples per candidate:
The pre-training pilot works well across all settings, staying within 0.11–0.68 pp of the best fixed β.