acceptodds
Under review as a conference paper at ICLR 2027

HO-FL: Hybrid-Order Federated Learning for Heterogeneous Edge Devices

Abstract

Federated learning (FL) on memory-constrained edge devices faces a dilemma: first-order (FO) optimization (i.e., backpropagation) demands substantial memory, whereas zeroth-order (ZO) optimization suffers from severe convergence slowdown. To resolve this dilemma, we introduce HO-FL, a hybrid-order FL framework that trains a model's bottom segment with ZO optimization and its top segment with FO optimization. Each device can flexibly select its order boundary according to its memory budget while participating in the training of the same global model. Moreover, our convergence analysis reveals a new, fundamental trade-off: clients with larger FO-trained segments can provide more accurate updates, but favoring them can underrepresent other clients' data. We connect this trade-off to the bias and variance of actual multi-step local updates, yielding a sampling optimization problem and a practical dimension-aware approximation with direct model averaging. Experiments on language tasks examine task performance, client memory, and sampling under data heterogeneity. The results show that hybrid-order local training can retain much of the full-FO performance with substantially lower client memory requirements. Our code is available at https://anonymous.4open.science/r/ICLR-HO-FL/.

Primary area:
optimization
Keywords:
Hybrid-Order Optimization, Federated Learning, Client Sampling

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_eAqGOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces HO-FL, a hybrid-order FL framework that trains a model’s bottom segment with ZO optimization and its top segment with FO optimization. Each device can flexibly select its order boundary according to its memory budget while participating in the training of the same global model. Moreover, their convergence analysis reveals a new, fundamental trade-off: clients with larger FO-trained segments can provide more accurate updates, but favoring them can underrepresent other clients’ data. The authors connect this trade-off to the bias and variance of actual multi-step local updates, yielding a sampling optimization problem and a practical dimension-aware approximation with direct model averaging. Experiments on language tasks examine task performance, client memory, and sampling under data heterogeneity. The results show that hybrid-order local training can retain the full-FO performance with lower client memory requirements.

Critical strengths

S1. The problem is critical for deploying large models on resource-constrained edge devices.

S2. The proposed framework demonstrates promising memory savings.

Critical weaknesses

W1. The paper should fucus on generalizing the results to edge federated learning beyond the evaluated configurations, e.g., by including more realistic hardware heterogeneity, additional workloads and adaptation settings, stronger non-IID scenarios, and varying client participation.

W2. The evaluation should clarify how memory budgets, order boundaries, data distributions, and client availability are assigned. It would be useful to evaluate the cases where resource availability and data distributions are correlated.

Questions for authors

Q1. Under what conditions does the dimension-aware sampling policy improve convergence to reach the uniform-client objective, considering the bias introduced by nonuniform sampling?

Q2. How closely does the proposed dimension-based score approximate the ideal sampling solution in Eqs.13-16?

Q3. Why does increasing \beta degrade performance under severe non-IID data, and can the policy detect when uniform sampling would be preferable?

Q4. How much of the reported benefit comes from heterogeneous order boundaries, as opposed to hybrid-order optimization or client sampling?

Q5. How sensitive are the results to resetting AdamW states at every communication round?

Q6. Do the reported memory savings translate into lower end-to-end training time and energy consumption on real-world heterogeneous edge devices?

5 comments
AuthorsOpenReview ↗

Response to Reviewer eAqG (Part 1/2)

We are grateful for the time you have taken to review our paper and for your insightful review. Below we address Q1–Q3 first, followed by Q4–Q6 and W1–W2 in our second reply.

Q1. Under what conditions does the dimension-aware sampling policy improve convergence to reach the uniform-client objective, considering the bias introduced by nonuniform sampling?

To identify this condition, we use Theorem C.9 in Appendix C.5. Substituting our sampling policy pi=(1−β)1N+βrip_i = (1-\beta)\frac{1}{N} + \beta r_i into Eq. 77 gives: 1T∑t=0T−1E∥∇f(wt)∥2≤CrateT1+(1−β)Du+βDRq+6ζ2β2(NK−1),\frac{1}{T}\sum_{t=0}^{T-1}\mathbb{E}\|\nabla f(w_t)\|^2 \le \frac{C_{\mathrm{rate}}}{\sqrt{T}}\sqrt{1 + \frac{(1-\beta)D_u + \beta D_R}{q}} + 6\zeta^2\beta^2\left(\frac{N}{K} - 1\right), where Du=1N∑idiZOD_u = \frac{1}{N}\sum_i d_i^{\mathrm{ZO}} and DR=1K∑i∈RdiZOD_R = \frac{1}{K}\sum_{i\in R} d_i^{\mathrm{ZO}}.

Compared against uniform sampling at β=0\beta = 0, non-uniform sampling yields a tighter convergence upper bound when: CrateT1+(1−β)Du+βDRq+6ζ2β2(NK−1)<CrateT1+Duq.\frac{C_{\mathrm{rate}}}{\sqrt{T}}\sqrt{1 + \frac{(1-\beta)D_u + \beta D_R}{q}} + 6\zeta^2\beta^2\left(\frac{N}{K} - 1\right) < \frac{C_{\mathrm{rate}}}{\sqrt{T}}\sqrt{1 + \frac{D_u}{q}}. We have added this discussion to Section 5 of our revision.

Q2. How closely does the proposed dimension-based score approximate the ideal sampling solution in Eqs.13-16?

Our dimension-based policy is a qualitative surrogate without a formal approximation-gap guarantee to Eq. (13). It upper-bounds the directional bias via Eq. (14), replaces dynamic variance-and-drift costs with static ZO dimensions, and restricts pp to the linear mixture in Eq. (16). This preserves the trade-off between lower ZO noise and uniform participation when ZO noise dominates over client drift, and our β\beta-sweeps in Table 4 confirm that this zero-overhead policy works well in practice.

Q3. Why does increasing β\beta degrade performance under severe non-IID data, and can the policy detect when uniform sampling would be preferable?

Setting a higher β\beta concentrates sampling on a small subset of clients with larger FO segments; under severe non-IID skew, this subset no longer represents the global data distribution and biases the aggregated model. Formally, Theorem 4.1 gives the bound O((1+1q∑ipidiZO)/T)+O(ζ2β2(N/K−1))\mathcal{O}\big(\sqrt{(1+\frac{1}{q}\sum_i p_i d_i^{\mathrm{ZO}})/T}\big) + \mathcal{O}\big(\zeta^2\beta^2(N/K-1)\big), where the first term captures ZO gradient variance and the second captures sampling representation bias. When data heterogeneity ζ2\zeta^2 is large, the ζ2β2\zeta^2\beta^2 bias term dominates, so increasing β\beta can hurt performance.

Our policy does not detect sampling bias online during training. Instead, Appendix D.2 provides pre-training selection guidelines, including a short pilot validation across β∈{0,0.25,0.5,0.75}\beta \in \{0, 0.25, 0.5, 0.75\}. To demonstrate its effectiveness, we conducted a pilot experiment on Qwen / AG News using 150 local steps and 2,400 samples per candidate:

DistributionSelected β\beta (3 seeds)Selected Acc (%)Best Fixed β\beta Acc (%)
IID[0.75, 0.5, 0.0]87.4387.91 (β=0.75\beta=0.75)
α=1.0\alpha=1.0[0.25, 0.75, 0.25]86.8687.26 (β=0.75\beta=0.75)
α=0.5\alpha=0.5[0.75, 0.5, 0.5]86.5886.69 (β=0.75\beta=0.75)
α=0.1\alpha=0.1[0.25, 0.5, 0.25]83.1583.83 (β=0.25\beta=0.25)

The pre-training pilot works well across all settings, staying within 0.11–0.68 pp of the best fixed β\beta.

AuthorsOpenReview ↗

Response to Reviewer eAqG (Part 2/2)

Q4. How much of the reported benefit comes from heterogeneous order boundaries, as opposed to hybrid-order optimization or client sampling?

To separate the gains of thess components, we compare four configurations:

MethodOPT / SST-2 IIDQwen / AG News (α=0.5\alpha=0.5)Qwen / SQuAD F1
DeComFL (Pure-ZO)51.95 ± 0.0055.83 ± 0.0873.14 ± 0.16
HO-FL (Uniform highest ZO boundary, β=0\beta=0)79.47 ± 0.1163.22 ± 1.4475.95 ± 0.05
HO-FL (Heterogeneous, β=0\beta=0)86.01 ± 0.1185.51 ± 0.4385.34 ± 0.03
HO-FL (Heterogeneous, β=0.75\beta=0.75)86.54 ± 0.2486.69 ± 0.5686.92 ± 0.10
  • Hybrid-order vs. pure-ZO (Row 1 vs. Row 2): Hybrid-order training provides a substantial accuracy jump over pure-ZO even under the uniform highest ZO boundary.
  • Utilizing heterogeneous resources (Row 2 vs. Row 3): Allowing each client to use its available memory instead of forcing all clients to the uniform highest ZO boundary yields large gains on all tasks.
  • Dimension-aware sampling (Row 3 vs. Row 4): Adding β=0.75\beta=0.75 provides further consistent improvements across all three tasks.

Q5. How sensitive are the results to resetting AdamW states at every communication round?

We evaluated resetting versus persisting local AdamW states on OPT / SST-2 with HO-FL:

β\betaLocal AdamW StateAccuracy (%)
0.0Reset per round (Default)86.01 ± 0.11
0.0Persistent local states85.67 ± 0.18
0.75Reset per round (Default)86.54 ± 0.24
0.75Persistent local states86.32 ± 0.30

On OPT/SST-2 IID, persisting local states shows no gain over resetting, while requiring extra per-client storage and state management for server replay.

W1 & Q6. Generalization beyond evaluated configurations and real-world end-to-end time/energy.

We agree that broader system tests would strengthen our empirical scope. However, this paper mainly focuses on designing HO-FL with convergence guarantees for memory-heterogeneous FL. Within this scope, achieving near-full-FO accuracy under heterogeneous memory limits is still a highly appealing property for edge scenarios. To demonstrate this feasibility, we measured peak memory, uplink payload, and per-step local compute time on Qwen2.5-1.5B (L=1024,E=5,q=2L=1024, E=5, q=2) across RTX 4090 24 GB, RTX 5090 32 GB, and A100 80 GB:

MethodPeak Mem (GiB) on A100Uplink (MB/client/round)4090 (ms/step)5090 (ms/step)A100 (ms/step)
FedAvg40.2614.358OOMOOM733.5
HO cut829.9613.113OOM983.6794.2
HO cut1619.5211.8681291.41034.8852.9
HO cut249.0800.6231356.41100.4913.2
DeComFL3.9170.0000581177.2821.41566.2

Since our experiments run serially on a single GPU rather than across parallel physical devices and real networks, end-to-end training time is not meaningful here. However, the above results show that FedAvg goes OOM on both 4090 and 5090, whereas HO-FL runs within each GPU's memory limit at comparable per-step latency. We added real-world edge deployment, time/energy profiling, and varying participation to Limitations and Future Work.

W2. The evaluation should clarify how memory budgets, order boundaries, data distributions, and client availability are assigned. It would be useful to evaluate the cases where resource availability and data distributions are correlated.

We have clarified our settings in Section 6.1 of the revision:

  • Memory budgets: The N=30N=30 clients are evenly partitioned into three fixed tiers of 10 clients each (Table 1), corresponding to per-device memory limits (e.g., 12/24/32 GiB on Qwen2.5-1.5B at L=1024L=1024).
  • Data distributions: Benchmarks use IID partitions, while AG News additionally evaluates Dir⁡(α)\operatorname{Dir}(\alpha) label skew (α∈{1,0.5,0.1}\alpha \in \{1, 0.5, 0.1\}) with 4,000 samples per client.
  • Client availability & sampling: All 30 clients are modeled as always available, with K=6K=6 clients sampled per round.

We agree resource–data correlation is a valuable future direction. Theoretically, Theorem 4.1 and Theorem C.9 already hold without assuming independence between resource and data distributions, because the bound ∥∑i(pi−1N)∇fi(w)∥2≤χ2(p)H(w)\left\|\sum_i (p_i - \frac{1}{N})\nabla f_i(w)\right\|^2 \le \chi^2(p)H(w) holds for arbitrary resource–data assignments. Thus, our framework extends naturally to correlated settings by selecting a smaller β\beta (as discussed in Appendix D.2) or incorporating tier-stratified sampling to control representation bias.

AuthorsOpenReview ↗

Gentle Reminder and Brief Summary

Dear Reviewer eAqG,

Thank you again for your time and thoughtful feedback. With the discussion deadline approaching, we wanted to share a quick recap of our responses:

  1. Theory & β\beta-Selection (Q1–Q3): Added explicit bound-improvement conditions in Section 5, clarified Eqs. (13–16) as a qualitative proxy, and added a low-cost β\beta-selection pilot within 0.11–0.68 pp of the best fixed β\beta.
  2. Ablations & AdamW State (Q4–Q5): Isolated the gains of hybrid-order updates, heterogeneous boundaries, and sampling, and confirmed that resetting AdamW states per round outperforms persistent states without extra state management.
  3. Cross-GPU Feasibility & Setup (W1, W2, Q6): Added memory, uplink, and step-time benchmarks across RTX 4090, RTX 5090, and A100, clarified our setup in Section 6.1, and noted edge deployment and resource–data correlation in Section 7.

We would be grateful to know whether these updates address your concerns. Thank you again for your consideration!

Best regards,

The Authors

Reviewer_4cBUOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes HO-FL, a hybrid-order federated learning framework that fine-tunes language models on memory-constrained, heterogeneous edge devices. The method combines zeroth-order optimisation for lower layers with first-order backpropagation for upper layers, letting clients choose optimisation boundaries based on their memory budgets. To reduce communication overhead, the framework uses seed-based reconstruction of zeroth-order updates and directly aggregates client updates. The paper provides a non-convex convergence analysis that characterises the trade-off between gradient-estimation variance and client data representation, motivating a dimension-aware client-sampling strategy. Experiments on OPT-125M, Qwen2.5-1.5B, and SmolLM3-3B across several language understanding and generation tasks demonstrate that HO-FL approaches full first-order federated learning performance while substantially reducing client memory requirements. Additional experiments investigate sampling preferences, data heterogeneity, and memory efficiency.

Critical strengths

-The paper addresses an important challenge in deploying and fine-tuning language models on heterogeneous edge devices. Allowing clients to independently configure their zeroth-order and first-order optimisation boundaries provides a flexible way to accommodate different memory capacities while maintaining a shared global model.

-The convergence analysis identifies a trade-off between reducing gradient estimation variance and preserving representation of heterogeneous client data. The proposed dimension-aware sampling strategy links this observation to a practical client selection mechanism, offering an optimisation perspective beyond conventional uniform federated sampling.

-Experiments across three language models and multiple downstream tasks show that HO-FL generally approaches full first-order performance while significantly outperforming the pure zeroth-order baseline. The reported peak memory reduction of up to 77.4%, along with experiments examining sampling preferences under varying levels of data heterogeneity, supports the framework's practical relevance.

Critical weaknesses

-The proposed local optimisation procedure closely follows existing hybrid-order approaches, particularly the surrogate objective based on boundary activation gradients used in HO-SFL. The primary differences are its integration into conventional federated learning, heterogeneous boundary selection, and client sampling. Although these extensions are practically relevant, the paper does not sufficiently isolate their methodological contributions through comparisons with straightforward federated adaptations of existing hybrid-order methods.

-The global objective assigns equal weight to all clients, but the proposed dimension-aware sampling prioritises clients with larger first-order segments while aggregating selected updates through unweighted averaging. This introduces systematic bias toward clients with greater computational resources, particularly under non-IID data distributions. Although the convergence analysis includes a representation-bias term, the practical consequences for underrepresented clients and the absence of importance correction require stronger justification.

-The empirical evaluation emphasises peak memory reduction and final task performance but provides limited evidence regarding end-to-end training time, client computation, and actual communication costs. Zeroth-order estimation requires multiple perturbed forward passes, while seed-based reconstruction transfers additional computation to the server. Without comprehensive comparisons of wall-clock convergence, total transmitted bytes, and server-side overhead, the overall efficiency advantages remain insufficiently established.

-The experiments simulate 30 clients using three predefined boundary tiers, while most tasks are evaluated under IID partitions. The non-IID analysis is restricted to AGNews, and the stronger label-skew setting shows that aggressive dimension-aware sampling can reduce performance and increase variability. Moreover, experiments use LoRA fine-tuning and simulated memory constraints rather than deployment on heterogeneous physical devices. These limitations weaken the evidence for robustness and practicality in real-world edge federated learning.

Questions for authors

-What are the key algorithmic differences between HO-FL and a direct federated implementation of existing hybrid-order optimisation methods such as HO-SFL? Can the authors provide controlled comparisons that separately quantify the benefits of heterogeneous order boundaries, seed-based update reconstruction, and dimension-aware sampling?

-How does the proposed non-uniform sampling strategy affect convergence to the original uniformly weighted global objective, particularly when clients with different memory capacities have substantially different data distributions? Can the authors quantify client-level performance disparities and compare the proposed aggregation scheme with importance-weighted aggregation approaches that correct for non-uniform participation?

  • Can the authors provide end-to-end wall-clock training time, communication volume, and server-side reconstruction costs compared with full first-order and pure zeroth-order federated learning? Additionally, how does HO-FL perform on physically heterogeneous edge devices and under more realistic combinations of non-IID data, client availability, and memory constraints?
3 comments
AuthorsOpenReview ↗

Response to Reviewer 4cBU (Part 1/2)

We thank Reviewer 4cBU for the constructive review and positive remarks on our problem motivation, theoretical trade-off, and memory savings. Below we address your questions in two parts.

W1 & Q1.1: What are the key algorithmic differences between HO-FL and a direct federated implementation of existing hybrid-order optimisation methods such as HO-SFL?

Beyond a direct federated implementation of HO-SFL, HO-FL introduces three key innovations:

  1. Heterogeneous boundary design: Clients independently choose order boundaries matching their memory budgets while averaging updates in the shared parameter space.
  2. Unified convergence bound: Theorem 4.1 establishes convergence under heterogeneous boundaries and non-uniform sampling, reducing to prior homogeneous-boundary, uniform-sampling rates (such as HO-SFL) as special cases (Eq. 12).
  3. Dimension-aware sampling: Grounded in our bound, a dependent-rounding sampling policy (Section 5) balances ZO gradient variance against representation bias.

Q1.2: Can the authors provide controlled comparisons that separately quantify the benefits of heterogeneous order boundaries, seed-based update reconstruction, and dimension-aware sampling?

To isolate heterogeneous boundaries and sampling, we compare uniform highest-ZO vs. heterogeneous boundaries:

MethodOPT / SST-2 IIDQwen / AG News (α=0.5\alpha=0.5)Qwen / SQuAD F1
HO-FL (Uniform highest ZO boundary, β=0\beta=0)79.47 ± 0.1163.22 ± 1.4475.95 ± 0.05
HO-FL (Heterogeneous, β=0\beta=0)86.01 ± 0.1185.51 ± 0.4385.34 ± 0.03
HO-FL (Heterogeneous, β=0.75\beta=0.75)86.54 ± 0.2486.69 ± 0.5686.92 ± 0.10

Comparing Row 1 vs. Row 2 shows the gains from heterogeneous boundaries, while Row 2 vs. Row 3 isolates dimension-aware sampling.

Seed-based update reconstruction is mathematically equivalent to dense bottom-segment aggregation and serves solely to reduce uplink communication. By transmitting only Eq=10Eq = 10 scalars instead of the full ZO delta each round, it saves roughly 2.39 GB on Qwen2.5-1.5B over 160 rounds at K=6K=6.

W2 & Q2.1: How does the proposed non-uniform sampling strategy affect convergence to the original uniformly weighted global objective, particularly when clients with different memory capacities have substantially different data distributions?

Substituting our policy pi=(1−β)1N+βrip_i = (1-\beta)\frac{1}{N} + \beta r_i into Theorem C.9 under aHβ2(NK−1)≤112a_{\mathrm{H}}\beta^2(\frac{N}{K}-1) \le \frac{1}{12} yields the convergence bound for the uniformly weighted objective f(w)f(w): 1T∑t=0T−1E∥∇f(wt)∥2≤CrateT1+(1−β)Du+βDRq+6ζ2β2(NK−1),\frac{1}{T}\sum_{t=0}^{T-1}\mathbb{E}\|\nabla f(w_t)\|^2 \le \frac{C_{\mathrm{rate}}}{\sqrt{T}}\sqrt{1 + \frac{(1-\beta)D_u + \beta D_R}{q}} + 6\zeta^2\beta^2\left(\frac{N}{K} - 1\right), where Du=1N∑idiZOD_u = \frac{1}{N}\sum_i d_i^{\mathrm{ZO}} and DR=1K∑i∈RdiZOD_R = \frac{1}{K}\sum_{i\in R} d_i^{\mathrm{ZO}}.

Non-uniform sampling lowers the first term via (1−β)Du+βDR<Du(1-\beta)D_u + \beta D_R < D_u at the cost of a bounded residual bias 6ζ2β2(NK−1)6\zeta^2\beta^2(\frac{N}{K}-1), giving a tighter bound than β=0\beta=0 when DR≪DuD_R \ll D_u and ζ2\zeta^2 is moderate relative to TT. Moreover, Theorem C.9 bounds selection skew via ∥∑i(pi−1N)∇fi(w)∥2≤β2(NK−1)H(w)\left\|\sum_i(p_i-\frac{1}{N})\nabla f_i(w)\right\|^2 \le \beta^2(\frac{N}{K}-1)H(w) without assuming resource–data independence, so it still holds when memory tiers have strongly different data distributions. We have added this discussion to our revision.

AuthorsOpenReview ↗

Response to Reviewer 4cBU (Part 2/2)

Q2.2: Can the authors quantify client-level performance disparities and compare the proposed aggregation scheme with importance-weighted aggregation approaches that correct for non-uniform participation?

Our experiments evaluate the shared global model on a single global test set, rather than on separate client-specific test sets. Consequently, the current evaluation does not quantify client-level performance disparities.

Direct averaging (Eq. 10) uses ΔDirect=1K∑i∈StΔi,t\Delta_{\mathrm{Direct}} = \frac{1}{K}\sum_{i\in S_t}\Delta_{i,t}, whereas Importance-Weighted (IW) aggregation uses ΔIW=1N∑i∈StΔi,tKpi\Delta_{\mathrm{IW}} = \frac{1}{N}\sum_{i\in S_t}\frac{\Delta_{i,t}}{Kp_i}. For an under-sampled ZO-heavy client with pi=1−βNp_i = \frac{1-\beta}{N}, IW applies a weight of 1K(1−β)\frac{1}{K(1-\beta)} instead of 1K\frac{1}{K}, which removes selection skew in expectation but inflates its already high ZO variance by 1(1−β)2\frac{1}{(1-\beta)^2} (up to 16×16\times at β=0.75\beta=0.75). On OPT / SST-2 IID, Direct averaging consistently outperforms IW with lower variance:

β\betaDirect Averaging (%)Importance-Weighted (%)Δ\Delta (IW − Direct, pp)
0.2586.31 ± 0.1886.20 ± 0.18-0.11
0.5086.39 ± 0.0785.82 ± 0.29-0.57
0.7586.54 ± 0.2486.09 ± 0.43-0.45

W3 & Q3.1: Can the authors provide end-to-end wall-clock training time, communication volume, and server-side reconstruction costs compared with full first-order and pure zeroth-order federated learning?

We agree with the reviewer on the importance of quantifying system overhead. On Qwen2.5-1.5B (L=1024,E=5,q=2L=1024, E=5, q=2), we measured exact peak memory, serialized uplink payload, and per-step local computation time across three heterogeneous GPUs (RTX 4090 24 GB, RTX 5090 32 GB, and A100 80 GB):

MethodPeak Mem (GiB)Uplink (MB/client/round)4090 (ms/step)5090 (ms/step)A100 (ms/step)
FedAvg40.2614.358OOMOOM733.5
HO cut829.9613.113OOM983.6794.2
HO cut1619.5211.8681291.41034.8852.9
HO cut249.0800.6231356.41100.4913.2
DeComFL3.9170.0000581177.2821.41566.2

Because our experiments are simulated serially on one individual GPU rather than across parallel physical edge devices and real networks, an end-to-end wall-clock time from serial simulation would not be meaningful. Nevertheless, these cross-GPU measurements provide a clear reference: FedAvg runs out of memory on both 24 GB RTX 4090 and 32 GB RTX 5090, whereas HO-FL runs smoothly within each GPU's memory limit at comparable per-step latency.

For server-side reconstruction, replaying E=5,q=2E=5, q=2 ZO updates and aggregating across K=6K=6 clients on Qwen2.5-1.5B takes only 222.50 ± 4.33 ms/round on A100, with exact numerical fidelity. This replay overhead is very small because it mainly consists of direction regeneration and simple vector additions rather than any forward or backward passes.

W4 & Q3.2: How does HO-FL perform on physically heterogeneous edge devices and under more realistic combinations of non-IID data, client availability, and memory constraints?

Due to the lack of physical heterogeneous edge devices, we have not yet evaluated HO-FL under real-world hardware deployments. However, achieving near-full-FO performance under heterogeneous memory constraints makes HO-FL highly appealing for edge scenarios. We have explicitly added real-world edge deployment, resource–data correlation, and varying client availability to Limitations and Future Work (Section 7) in our revision.

AuthorsOpenReview ↗

Gentle Reminder and brief Summary

Dear Reviewer 4cBU,

Thank you again for your thoughtful review and positive assessment of our paper. Before the discussion window closes, we wanted to share a brief summary of our updates:

  1. Novelty, Ablations & Seed Replay (W1, Q1): Clarified our three innovations beyond HO-SFL, isolated the gains of heterogeneous boundaries and dimension-aware sampling.
  2. Sampling Bias & Importance Weighting (W2, Q2): Derived our policy's explicit bound on the uniform global objective and showed that direct averaging outperforms importance-weighted aggregation with lower variance.
  3. Cross-GPU Overhead & Edge Scope (W3, W4, Q3): Added peak memory, uplink payload, server replay latency, and step-time benchmarks across RTX 4090, RTX 5090, and A100.

We would be grateful to know whether these updates address your remaining concerns. Thank you again for your support and time!

Best regards,

The Authors

Reviewer_76MCOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents HO-FL, a method that combines first-order and zeroth-order updates to enable memory-efficient LLM fine-tuning in FL. HO-FL hinges on two factors. First, the bottom layers of the LLM are trained without backprop, using only ZO updates, while the top layers are trained with FO optimization. The boundaries are chosen based on client-specific memory capabilities. Second, clients are sampled probabilistically using a knob β\beta based on the number of parameters they update with FO vs. ZO optimization. A larger β\beta close to 1 pushes the sampling more towards clients with greater FO optimization capabilities. Combined, these elements help HO-FL outperform pure ZO methods while bringing performance closer to pure FO methods. These results persist across different model backbones and tasks, demonstrating its generality in multiple settings.

Critical strengths
  1. Combination of Empirical and Theoretical Analysis - The method provides convergence bounds as well robust empirical proof of its effectiveness. The combination of these factors make HO-FL more compelling. The use of multiple model sizes ranging from 125M-3B parameters is especially appreciated.
  2. Clear Method Details - Section 3 is well-written and provides effective details for understanding the method. The section highlights both client-side and server-side actions and disambiguates how the combination of ZO and FO is done.
Critical weaknesses
  1. Missing Baselines/Novelty - The method seems to be a somewhat straightforward application of HO optimization to FL. Prior methods like ZO-Bench and LoHO seem to propose something quite similar in the centralized setting. The paper also mentions other FL split-learning versions of similar concepts. It's unclear how much the paper innovates on these prior baselines. Similarly, some of the baselines could be applies to regular FL without much adaptation making them more representative baselines than DeComFL.
  2. Missing Ablations - The method claims that the HO optimization is responsible for the improvement over DeComFL. However, it does not provide an ablation where only the top layers are trained with FO and the remaining layers are frozen. It similarly does not provide an ablation where all layers/most layers are trained with FO but with a smaller LoRA rank (e.g. r=4) to accommodate clients' memory constraints.
  3. Experimental Settings - Even at β=0\beta=0, the mean round total memory (GiB) relative to DeComFL is still almost 5x. It increases to almost 7x at β=0.75\beta=0.75 which is the preferable setting where the method gets closer to the performance of full FO optimization. The performance gains seem to come at a pretty substantial memory tradeoff.
Questions for authors
  1. Is there a reason at higher heterogeneity, lower β\beta is preferable? I don't have intuition for why this would be true since it would seem like a larger β\beta where you prioritize clients with more expressive FO updates should generally work a lot better.
4 comments
AuthorsOpenReview ↗

Response to Reviewer 76MC (Part 1/2)

We thank Reviewer 76MC for the constructive feedback and for recognizing our theoretical bounds and multi-scale empirical validation. Below we address your concerns in two parts.

W1: Novelty and comparison with prior hybrid-order baselines

In both Section 2 and Appendix A, we summarize existing HO methods and acknowledge their contributions. Compared to prior work, our main contributions are threefold:

  1. Heterogeneous boundary design: We bring HO into FL and introduce heterogeneous order boundaries matched to each device's memory budget.
  2. Unified convergence bound: We derive the non-convex convergence rate under heterogeneous boundaries and non-uniform sampling (Theorem 4.1), which generalizes and reduces not only to centralized rates (FedAvg, pure ZO-FL) but also to prior split-learning HO bounds (such as HO-SFL) as special cases.
  3. Dimension-aware sampling: Grounded in our convergence bound, we formulate a practical sampling policy via dependent rounding that balances ZO gradient variance against representation bias.

Importantly, Contributions 1 and 3 are general and can be applied to any federated HO method. To verify their effectiveness, we implemented federated LoHO as a representative baseline. Under our taxonomy in Appendix A.2, LoHO falls under the Exact End-to-End Evaluation surrogate, whereas HO-FL adopts the First-Order Activation Feedback surrogate. We compare the naive FL version of LoHO, LoHO with heterogeneous boundaries, LoHO with both heterogeneous boundaries and dimension-aware sampling, and HO-FL:

MethodOPT / SST-2 IIDQwen / AG News (α=0.5\alpha=0.5)Qwen / SQuAD F1
LoHO (Uniform tightest boundary, β=0\beta=0)79.32 ± 0.0763.31 ± 1.2875.97 ± 0.10
LoHO (Heterogeneous boundaries, β=0\beta=0)85.93 ± 0.2485.34 ± 0.4385.26 ± 0.19
LoHO (Heterogeneous boundaries, β=0.75\beta=0.75)86.24 ± 0.1386.53 ± 0.3886.64 ± 0.10
HO-FL (β=0.75\beta=0.75)86.54 ± 0.2486.69 ± 0.5686.92 ± 0.10

Comparing Rows 1 to 3 confirms that adding heterogeneous boundaries and dimension-aware sampling yields consistent gains across all tasks. Moreover, while LoHO (β=0.75\beta=0.75) reaches accuracy comparable to HO-FL (β=0.75\beta=0.75), its Exact End-to-End Evaluation surrogate requires extra forward passes through the top FO segment for ZO updates, whereas HO-FL confines all ZO perturbations strictly to the bottom segment.

W2: Ablations (Top-FO-Only and Small-Rank FO-LoRA)

Thank you for suggesting these important baselines. We added both requested ablations:

  • Freezing bottom layers instead of updating them with ZO drops accuracy on Qwen2.5-1.5B, confirming the effectiveness of ZO updates:
MethodAG News (α=0.5\alpha=0.5) (%)SQuAD F1 (%)
Top-FO-Only (Freeze Bottom, β=0.75\beta=0.75)83.37 ± 0.2183.65 ± 0.22
HO-FL (β=0.75\beta=0.75)86.69 ± 0.5686.92 ± 0.10
  • On Qwen2.5-1.5B, although reducing the LoRA rank from r=8r=8 to r=4r=4 reduces the number of trainable parameters, backpropagation still requires caching intermediate activations, leaving peak memory nearly unchanged at about 40 GiB. In contrast, HO-FL avoids activation caching on the ZO layers, cutting peak memory to 9 GiB on Tier 3 devices:
MethodQwen Peak Mem (GiB)Qwen / SQuAD F1
FedAvg FO-LoRA (r=8r=8)40.26187.59 ± 0.07
Small-Rank FO-LoRA (r=4r=4)40.24187.00 ± 0.07
HO-FL (β=0.75\beta=0.75)9.080–29.96186.92 ± 0.10
AuthorsOpenReview ↗

Response to Reviewer 76MC (Part 2/2)

W3: Experimental settings and memory tradeoff relative to DeComFL

We clarify that although the mean round-total memory is higher than DeComFL, this does not require any individual client to exceed its local memory budget. While DeComFL keeps memory very low, its pure-ZO updates suffer from large accuracy drops across tasks. Our goal is to offer a practical middle ground between pure ZO and full FO under heterogeneous memory budgets: HO-FL lets low-memory devices join training within their local limits instead of being blocked by full-FO memory demands, while reaching near-full-FO accuracy. Moreover, β\beta is a tunable hyperparameter rather than a fixed cost, so practitioners can simply choose a smaller β\beta (such as 0 or 0.25) when lower aggregate round memory is preferred and still retain strong performance:

MethodQwen Peak Mem (GiB)Qwen / AG News (α=0.5\alpha=0.5)
DeComFL (Pure-ZO)3.91755.83
HO-FL (β=0\beta=0)9.080–29.96185.51
HO-FL (β=0.75\beta=0.75)9.080–29.96186.69
FedAvg (Full-FO)40.26188.15

Q1: Why lower β\beta is preferable at higher heterogeneity

To better illustrate this intuition, consider a simple scenario under strong non-IID partitioning. Severe non-IID splits, whether Dirichlet-based or pathological, naturally leave individual clients dominated by a subset of classes. For instance, in a binary cat-vs-dog classification task, suppose Client A happens to hold mostly cat images while Client B holds mostly dog images, and Client A has more memory to support more FO layers. Setting a high β\beta concentrates sampling on the high-FO subset such as Client A, which biases the global model toward predicting cat. Therefore, a higher β\beta can hurt performance when the data is more heterogeneous.

Formally, this trade-off is characterized in Theorem 4.1, where the convergence bound is O((1+1q∑ipidiZO)/T)+O(ζ2β2(N/K−1))\mathcal{O}\big(\sqrt{(1+\frac{1}{q}\sum_i p_i d_i^{\mathrm{ZO}})/T}\big) + \mathcal{O}\big(\zeta^2 \beta^2 (N/K - 1)\big). Here, the first term captures the ZO gradient-estimation variance, and the second term captures the sampling representation bias. Increasing β\beta reduces the first term while increasing the second term. Under IID or mild skew where ζ2\zeta^2 is small, the first term dominates, so a larger β\beta tends to be beneficial. Under severe skew where ζ2\zeta^2 is large, the second term dominates and raises the risk of representation bias, making a smaller β\beta safer to keep this bias controlled.

Thank you for your thorough response. Based on the additional ablations and comments, I have increased my score to a 3.

Thank you so much for your time and your constructive comments. We are glad that our responses addressed your concerns.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal and Discussion

Dear Area Chairs,

We thank you for your time and effort in reviewing our submission. After our rebuttal and revised manuscript, Reviewer 76MC and eAqG both increased their ratings from 2 to 3. Reviewer 4cBU (rating 3) is still pending a response. All reviewers now hold positive evaluations of our submission.

We are encouraged by the reviewers' recognition of our work's strengths:

  • Practical importance for deploying and fine-tuning LLMs on memory-constrained edge devices (76MC, 4cBU, eAqG)
  • Flexible heterogeneous-boundary design delivering substantial peak memory savings (4cBU, eAqG)
  • Unified convergence theory characterizing the variance–representation trade-off and guiding sampling (76MC, 4cBU)
  • Clear method presentation and multi-scale empirical validation across 125M–3B models (76MC, 4cBU)

Manuscript Revisions (Highlighted in Blue)

  • Theoretical clarifications: We clarified how HO-FL's execution paradigm differs from centralized and split-learning HO methods (Section 2, Appendix A.2), and derived the explicit finite-budget convergence bound for our sampling policy from Theorem C.9 (Section 5, Eq. 17).
  • Experimental additions: We clarified our memory-tier and sampling protocol (Section 6.1) and added controlled ablations isolating heterogeneous boundaries from dimension-aware sampling (Section 6.3, Appendix E.6, Table 9).
  • Limitations and future work: We expanded Section 7 to acknowledge physical-hardware time and energy profiling, correlated resource–data distributions, and time-varying client availability.

Key Concerns Resolved via Additional Experiments and Analyses

  • Method generality and baseline necessity: We resolved questions on novelty and alternative memory-saving baselines by showing that our heterogeneous boundaries and sampling generalize to federated LoHO, while Top-FO-Only and Small-Rank LoRA (r=4r=4) fail to match HO-FL's accuracy or memory reduction.
  • Sampling bias and optimizer design: We justified direct averaging over importance weighting, validated a low-cost pre-training β\beta-selection pilot, and confirmed that resetting AdamW states per round preserves accuracy without state-management overhead.
  • System overhead and hardware feasibility: We quantified uplink compression, server replay latency, and per-step compute times across RTX 4090, RTX 5090, and A100 GPUs, demonstrating that HO-FL runs smoothly within tighter memory budgets where full-FO FedAvg goes OOM.

We believe these updates have substantially strengthened the paper and addressed the reviewers' core concerns. Thank you again for your consideration.

Sincerely,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.