acceptodds
Desk rejected at ICLR 2027

Fed3DLoRA: Cross-Layer Tensor Consensus for Heterogeneous Federated Fine-Tuning

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Federated fine-tuning with low-rank adaptation (LoRA) has to accommodate clients whose resources support adapters of different ranks. Existing methods combine such adapters one weight matrix at a time, although every layer of a transformer reads from the same residual stream, which couples the updates of a projection across depth. Fed3DLoRA exploits this cross-layer structure: the server stacks the updates along depth into a tensor, builds a low-dimensional coordinate system shared across layers and clients, and expresses every client’s update, whatever its rank, as a compact core in it. This makes a client-specific, direction-dependent consensus tractable: cores are combined under metrics derived from the updates themselves, while everything outside the shared space is kept as the plain mean before the rank-constrained write-back. Aggregation runs entirely on the server and leaves client computation and per-round communication unchanged. We prove that it is invariant to how clients factorise their adapters, reduces to plain averaging when the client metrics coincide, and admits a convergence bound that accounts for the rank truncation. Extensive experiments on diverse LLM fine-tuning tasksExtensive experiments on diverse LLM fine-tuning tasks show that Fed3DLoRA outperforms existing methods in accuracy under joint rank and data heterogeneity, and that cross-layer consensus improves on its layer-wise counterpart in every setting tested show that Fed3DLoRA outperforms existing methods in accuracy under joint rank and data heterogeneity, and that cross-layer consensus improves on its layer-wise counterpart in every setting tested.

Primary area:
foundation or frontier models, including LLMs
Keywords:
federated learning, parameter-efficient fine-tuning, low-rank adaptation, rank heterogeneity

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_jQcqOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes Fed3DLoRA, a server-side aggregation method for federated LoRA with heterogeneous client ranks and data distributions. The method first corrects each client update against the adapter prefix it actually received, then stacks updates from the same projection family across Transformer layers into a third-order tensor. Shared layer, input, and output bases are used to obtain compact client cores, and a client-specific metric based on update energy and direction is used to compute a cross-layer consensus. The consensus is applied only inside the shared subspace, while the product-space mean is kept in the orthogonal complement, followed by a rank-constrained spectral write-back for clients with different ranks. The paper proves gauge invariance, zero-update consistency, reduction to averaging under identical metrics, a bounded deviation from averaging, and a conditional stationarity result. Experiments on Commonsense8 and MultiNLI with Qwen3 and Llama backbones show that Fed3DLoRA is competitive with existing heterogeneous federated fine-tuning methods and generally improves over layer-wise and product-space averaging controls.

Critical strengths

Strength 1: Fed3DLoRA jointly models client updates across layers instead of aggregating each layer independently.

Strength 2: The reference-correction step cleanly handles clients with different LoRA ranks.

Strength 3: The method keeps client-side computation and communication unchanged while improving server-side aggregation.

Critical weaknesses

Weakness 1: Fed3DLoRA assumes that updates from different Transformer layers can be meaningfully placed in one shared tensor space. The paper uses common input and output bases across layers, yet it is not clear why update directions from different depths should be comparable in the same coordinates.

Weakness 2: The client metric is built from the update magnitude and direction. This makes the aggregation rule depend heavily on how large a client update is. A large update may carry useful information, or it may come from client drift or noisy local training. The paper does not show why these quantities are reliable signals for client importance, and the chosen metric form is not directly connected to the federated objective.

Weakness 3: The convergence analysis mainly shows that the method remains controlled when its update stays close to product-space averaging. It does not show that the proposed cross-layer consensus gives a better estimate of the global update, reduces client drift, or leads to a better optimization direction. This leaves a gap between the motivation of the method and what the theory actually proves.

Weakness 4: The aggregation rule depends on the specific basis used inside the shared subspace. The diagonal metric is not invariant to rotations of this basis, so two different orthonormal bases spanning the same subspace can produce different client metrics and different aggregated updates. This makes the aggregation sensitive to the basis choice and gives no guarantee that the result is determined by the shared subspace itself.

Questions for authors

Question 1: Why should updates from different Transformer layers share the same input and output bases, and under what conditions is this cross-layer representation expected to hold?

Question 2: Why are update magnitude and direction used to define client importance in the metric? How does this rule distinguish useful task information from client drift or noisy local updates?

Question 3: Can the authors show that the proposed consensus gives a better estimate of the global update or reduces client drift, instead of only bounding its deviation from product-space averaging?

Question 4: Since the diagonal metric is not invariant to rotations within the shared subspace, how can the method guarantee stable aggregation when different equivalent bases can produce different aggregated updates?

7 comments

Response to Reviewer jQcq (part 1 of 4)

We thank the reviewer for a precise review. The four observations are mathematically correct, and the paper does not claim otherwise; where it was not explicit enough about them, it should have been. Our answers therefore do not dispute the observations. For each, we state exactly what the construction guarantees, quantify on recorded uploads and in end-to-end runs how much the property matters, and name the additions we will make to the text.

Q1 / W1. Why should updates from different layers share input and output bases, and when is this expected to hold?

Three parts. (i) Structure. The input mode of a projection family is the residual stream: every layer's query and value projections read the same hidden-state coordinates, so an input basis UIU_I shared across layers is a basis of a space the layers do share. The output mode (per-layer head coordinates) and the layer mode are not shared by construction. There, the shared bases are not an assumption about the updates but a fitted coordinate system: the leading eigenvectors of the cohort's energy-weighted Gram operators (Eq. 4), i.e. the coordinates that capture the most update energy for the prescribed dimension, re-estimated in every round. We do not assume that updates at different depths are similar; we fit the cohort's common coordinates and measure what they capture. (ii) What happens when the representation is poor. The consensus acts only inside the shared subspace; the orthogonal complement is aggregated by the product-space mean (Prop. 3), and the deviation from the mean is bounded (Prop. 4). A shared subspace that captures little energy therefore makes Fed3DLoRA approach product-space averaging; it cannot move it away from averaging by more than the bound. (iii) When it holds, observably. The operative condition is the captured energy fraction, which the server computes in every round: with (cL,cO,cI)=(16,128,128)(c_L, c_O, c_I) = (16, 128, 128) the shared subspace retains medians of 32–65% of the cohort's weighted update energy across settings and rounds (Appendix E.4), so the consensus acts on roughly half of the update and averaging on the rest. That fraction is not carried by a few dominant layers. Measured layer by layer on the 36 recorded events of all four settings (72 event–families, 2,016 layer records; output and input bases only, no layer-mode basis), the cohort-fitted bases capture a median of 32–73% of each layer's update energy, 26–167 times what a random subspace of the same dimensions captures; in every event at least 20 of the 28 layers, and typically 26–28, exceed ten times the random level, and the layer with the lowest capture in an event still lies several times above it (3–25% at rounds 5 and 10; in the setting we examined in detail these are the last two layers, holding under 0.5% of the family's update energy). Every layer therefore has a substantial component in the shared coordinates, and what falls outside them is averaged (Prop. 3). Two further measurements indicate that the shared coordinates are informative rather than arbitrary. Cores of clients with the same task-pair profile are closer than cores of clients with different profiles in all 36 recorded event–family combinations (Figure 6). And a new control that applies the joint metric on layer-wise bases (separate bases per layer with the same total number of core coordinates, cores concatenated) is worse than Fed3DLoRA by 0.17 / 5.43 / 3.40 / 0.67 accuracy points on CS8 / Qwen3-0.6B, CS8 / Qwen3-1.7B, MultiNLI / Qwen3-0.6B and MultiNLI / Qwen3-1.7B (three seeds each, paired; lower choice NLL for Fed3DLoRA in every setting), whereas the consistent layer-wise control is behind by only 0.30 / 3.56 / 0.11 / 0.84: a joint metric is useful only in shared coordinates. In the revision, §3.2 will state (i)–(iii) explicitly and name the energy fraction as the condition under which the cross-layer representation is doing work (with the per-layer capture table in Appendix E.4), and the abstract and introduction will present the shared residual stream as the reason the layers can be given common coordinates, not as a claim that their updates are similar.

Response to Reviewer jQcq (part 2 of 4)

Q2 / W2. Why magnitude and direction define the metric; how the rule treats drift or noise; the link to the federated objective.

First, a correction of the premise. The metric does not depend on how large a client's update is: qkq_k is the relative energy profile (squared coordinates divided by their mean, clipped at cc) and vkv_k is a unit vector, so MkM_k is invariant to gk→s,gkg_k \to s\\, g_k for every s>0s > 0. A client's magnitude enters the consensus only through gkg_k itself in ∑kwkMkgk\sum_k w_k M_k g_k, exactly as it enters averaging. What the metric encodes is where (which coordinates) and along which direction a client's update lies, not how big it is. Two controls confirm that magnitude is not what the gain comes from: the norm-weighted rule ak∝wk∣Dk∣Fa_k \propto w_k \\|D_k\\|_F, the literal "importance = size" rule trained under the same protocol, is worse than Fed3DLoRA in all four settings (by 0.31–1.22 points, with higher choice NLL in each); and rescaling the Fed3DLoRA update to the norm of the averaging update in every round (Fed3DLoRA-NM, Table 5) keeps the accuracy gain over averaging and lowers choice NLL further in every seed.

Link to the objective. Eq. (8) is the minimiser of ∑kwk(g−gk)⊤Mk(g−gk)\sum_k w_k (g - g_k)^{\top} M_k (g - g_k). Read as an objective, this replaces each client's local loss by a quadratic model around that client's own update, with curvature MkM_k, and takes the minimiser of the weighted sum of these models. Mk=λI+Diag(qk)+μ,vkvk⊤M_k = \lambda I + \mathrm{Diag}(q_k) + \mu\\, v_k v_k^{\top} is a damped, normalised diagonal-plus-rank-one proxy of the client's empirical Fisher information in the shared coordinates (the outer product of the client's own update direction, and its diagonal), so the rule is Fisher-weighted merging (Matena & Raffel, 2022; cited in §5), applied per round in the cross-layer coordinates and with the complement averaged. It is a proxy: it is built from the client's aggregated update rather than from per-example gradients, and we do not claim it equals the Fisher. We will add this derivation to §3.3, which currently motivates the two terms only geometrically.

Drift and noise. We do not claim that the metric identifies drift, and we will not claim it in the revision. What the rule does with a client that disagrees with the others is characterised rather than assumed: Prop. 5 gives the closed form of the rank-one consensus, in which a direction shared by no other client is scaled by (λ+μ)/(λ+μwk)(\lambda + \mu)/(\lambda + \mu w_k), a bounded factor; Prop. 4 bounds the total deviation from the mean by χ(∑kwk∣P(Dk−Dˉ)∣2)1/2\chi \big( \sum_k w_k \\|P(D_k - \bar{D})\\|^2 \big)^{1/2}; and the guard removes the diagonal term for updates at machine-epsilon scale. That the quantities the metric uses are informative rather than arbitrary is tested by the shuffled-metric control (Table 3), which keeps every metric's spectrum but breaks its alignment with the client's update and loses 0.1–1.8 points, and by the fixed-input evidence under Q3.

Response to Reviewer jQcq (part 3 of 4)

Q3 / W3. Does the consensus give a better estimate of the global update, or only a bounded deviation?

We agree on what Theorem 1 shows and does not show. It is a stability result: the iterates stay within a bounded distance of product-space local SGD, and the bound reduces to that of averaging when the client metrics coincide. It is not an improvement result, and the revision will say so in those words in §3.5 and in the conclusion. Nor do we claim that the consensus is a closer estimate of the full-participation update than the cohort mean is: the cohort mean is the natural estimator of that quantity, and the consensus departs from it inside the shared subspace by design (Props. 3–4). What we claim is that this departure improves the optimisation outcome, and that claim is empirical. We have three kinds of evidence; the first is on identical inputs, where no training-trajectory confound exists. (a) Fixed-input one-step evaluation. On the 27 recorded events (3 settings × 3 seeds × rounds 1/5/10) we apply the consensus and the product-space mean of the same uploads to the same global adapter and score the result on the held-out diagnostic set, one projection family at a time with the other averaged. The consensus gives higher accuracy and lower choice NLL than the mean in every setting: medians over events of +0.20 / −0.0023 on CS8 / Qwen3-0.6B, +0.39 / −0.0098 on CS8 / Qwen3-1.7B and +0.10 / −0.0029 on MultiNLI / Qwen3-0.6B. Per event–family, one-step accuracy is above averaging in 37 of the 54 cases, equal in 3 and below in 14, and the count does not fall over training (11, 13 and 13 of 18 at rounds 1, 5 and 10). (b) Scalar steps cannot reproduce it. No global step on the mean, including the FedExP step, reaches Fed3DLoRA on CS8 / Qwen3-1.7B, which is above the mean in every seed (Table 5, Appendix D.3), and the norm-matched variant shows the gain is not a scale effect. (c) Every simplification of the rule is worse end to end. Layer-local coupling, diagonal-only and rank-one-only metrics, and four scalar-weighted rules (norm, similarity, core similarity, trace), each trained under the same protocol on the four settings of Table 3 with three seeds, are behind Fed3DLoRA in 14 of 16 accuracy comparisons for the scalar rules and in all 12 for the others (full tables in our responses to Reviewers NNmM and Zdmz). We will state the empirical status of the claim precisely in the revision.

Reviewer_ZdmzOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

Fed3DLoRA proposes a server-side aggregation method for federated LoRA fine-tuning with heterogeneous client ranks and data distributions. It constructs reference-corrected client updates, represents their cross-layer structure in shared Tucker subspaces, and computes a consensus using client-specific metrics derived from update energy and direction. The consensus replaces the mean within the shared subspace while preserving the mean in its complement, followed by rank-constrained write-back. Experiments on Commonsense8 and MultiNLI compare the method with heterogeneous federated LoRA baselines across several language model backbones.

Critical strengths
  1. A well-defined aggregation mechanism and consistency properties. The work develops a matrix-weighted consensus rule in a shared cross-layer update representation while retaining standard client-side LoRA training. Reference correction, gauge invariance, reduction to averaging under appropriate conditions, and complement-preserving reconstruction give the approach a clear mathematical basis.

  2. Targeted experimental controls and transparent cost reporting. The comparisons with layer-wise consensus, product-space averaging, and shuffled metrics examine distinct elements of the proposed design. The server-step and norm-matching experiments also address the possibility that some gains arise merely from a larger update magnitude. The paper reports numerical behavior and server-side aggregation times, making the benefits and costs easier to assess.

Critical weaknesses

1. The contribution of cross-layer metric coupling is not fully isolated from that of the shared representation.

Table 3 shows that the complete method outperforms the layer-wise consensus control by 0.23–0.95 percentage points across four settings, with matched total core dimensionality. This provides useful evidence for the joint cross-layer design. However, as the authors acknowledge, the comparison changes both the representation bases and the aggregation structure: Fed3DLoRA uses shared Tucker bases and a joint metric, whereas the control uses separate layer-wise bases and metrics. The observed gain therefore cannot be assigned specifically to cross-layer coupling.

A controlled variant that retains the same shared Tucker representation while restricting the metric to layer-local interactions would clarify the independent role of cross-layer coupling. A simpler cross-layer weighting scheme based on client update norms or similarities would provide another useful reference for evaluating the additional structure of the proposed metric. These comparisons are particularly relevant because the claimed advance lies in how cross-layer information changes aggregation decisions.

2. The necessity of the full metric formulation remains uncertain relative to its numerical and computational cost.

The metric combines a coordinate-wise energy term with a rank-one directional term. Appendix E.4 offers informative fixed-input diagnostics, but their separate contributions to end-to-end accuracy are not established. In Table 21, the diagonal energy term is inactive for all recorded client-family observations in later rounds of the MultiNLI / Qwen3-0.6B setting. The rule then reduces to the isotropic and rank-one terms for those observations. This suggests that full, diagonal-only, and rank-one-only variants should be compared under matched end-to-end training conditions to identify when the more elaborate metric is useful.

The activation of the diagonal term depends on an absolute threshold based on FP32 machine precision. A sensitivity analysis of threshold choice and update scale would help clarify how consistently the intended metric operates. The cost of the full procedure is also material: Appendix F reports server aggregation times approximately 2.6–3.3 times those of product-space averaging with the same dense-SVD write-back, and in some settings aggregation takes longer than the slowest participating client's local training. The authors already report useful server-step controls and disclose these costs; an accuracy-versus-runtime comparison of the metric variants would help establish the practical value of the full formulation.

Additional statistical and presentation observations (minor). The paper reports three-seed means and standard deviations, but some margins over strong baselines are small. On Commonsense8 / Qwen3-1.7B, the mean improvement over the strongest listed external baseline is 0.13 percentage points on a diagnostic set of 768 examples. Paired per-seed differences and, if feasible, a task-level breakdown would help assess the stability and distribution of the gains. Figure 2 includes small annotations and legends, Table 4 has dense numerical columns, and Figure 3 would benefit from clearer plot labels.

Questions for authors
  1. Cross-layer mechanism. With the shared Tucker bases held fixed, what is the effect of restricting metric coupling to layer-local information? This would distinguish gains from the shared representation and from cross-layer aggregation decisions.

  2. Metric necessity and numerical behavior. How do full, diagonal-only, and rank-one-only metrics compare in end-to-end accuracy and server time? How sensitive is the diagonal term's activation to the FP32-derived threshold and update scale?

  3. Practical trade-off. How does the accuracy–runtime trade-off compare with a lightweight cross-layer weighting rule based on update norms or similarities, under the same client protocol and write-back procedure?

4 comments

Response to Reviewer Zdmz (part 1 of 5)

We thank the reviewer for a careful and constructive review, and in particular for naming the controls that would settle the open questions: a shared-representation control with a layer-local metric, the diagonal-only and rank-one-only metrics, a sensitivity analysis of the guard, and a lightweight norm- or similarity-weighted rule. We have run all of them end to end, on the four settings of Table 3 with three seeds each (a separate batch with its own seeds; all comparisons are paired within it: same client data, rank binding, participation sequence and initialisation per seed), and we re-measured the server cost. No number in the main text changes; every table below will go into the revision.

Q1 / W1. Cross-layer coupling with the shared Tucker bases held fixed.

We ran the control the reviewer describes and its mirror image, so that the shared representation and the cross-layer metric can be attributed separately.

  • H1a, shared bases + layer-local metric. The shared Tucker bases of Fed3DLoRA are kept, but the metric and the consensus of Eq. (8) are solved separately within each of the cL=16c_L = 16 retained layer-mode slices of the core, so the aggregation decision for one slice cannot use the energy profile or direction of any other slice.
  • H1b, layer-wise bases + joint metric. Separate bases are fitted for every layer with the same total number of core coordinates (as in the layer-wise control), the per-layer cores are concatenated, and the joint metric consensus runs over the concatenation.
  • LW, layer-wise bases + layer-wise metrics, the control of Table 3, re-run in this batch.

Table Z1. Fed3DLoRA minus control, final round, mean over 3 seeds; accuracy points / choice NLL (positive accuracy and negative NLL favour Fed3DLoRA).

ControlCS8 / Qwen3-0.6BCS8 / Qwen3-1.7BMultiNLI / Qwen3-0.6BMultiNLI / Qwen3-1.7BH1a shared bases, layer-local metric+1.30 / −0.0172+2.86 / −0.1153+1.31 / −0.0161+0.62 / −0.0282H1b layer-wise bases, joint metric+0.17 / −0.0004+5.43 / −0.1672+3.40 / −0.0461+0.67 / −0.0239LW layer-wise bases and metrics+0.30 / −0.0083+3.56 / −0.1439+0.11 / −0.0188+0.84 / −0.0218\begin{array}{l|cccc} \text{Control} & \text{CS8 / Qwen3-0.6B} & \text{CS8 / Qwen3-1.7B} & \text{MultiNLI / Qwen3-0.6B} & \text{MultiNLI / Qwen3-1.7B} \\\\ \hline \text{H1a shared bases, layer-local metric} & \text{+1.30 / −0.0172} & \text{+2.86 / −0.1153} & \text{+1.31 / −0.0161} & \text{+0.62 / −0.0282} \\\\ \text{H1b layer-wise bases, joint metric} & \text{+0.17 / −0.0004} & \text{+5.43 / −0.1672} & \text{+3.40 / −0.0461} & \text{+0.67 / −0.0239} \\\\ \text{LW layer-wise bases and metrics} & \text{+0.30 / −0.0083} & \text{+3.56 / −0.1439} & \text{+0.11 / −0.0188} & \text{+0.84 / −0.0218} \end{array}

With the shared bases held fixed, cross-layer metric coupling is worth +0.62 to +2.86 points and lower choice NLL in every setting (H1a). With the joint metric held fixed, the shared bases are worth +0.17 to +5.43 points, again with lower NLL in every setting (H1b). The attribution the reviewer asks for is therefore: both components contribute, and they are complementary rather than additive. The H1a and H1b gaps do not sum to the LW gap, and in three of the four settings one of the two single-factor designs is worse than the consistent layer-wise control. This is consistent with the construction: the metric compares coordinates across clients, so coupling across layers is only meaningful in shared coordinates, and shared coordinates only pay off when the consensus uses them jointly. We will add Table Z1 next to Table 3 and replace the sentence of §4.3 that currently says the comparison does not isolate the two factors.

Response to Reviewer Zdmz (part 2,3 of 5)

Q2 / W2. End-to-end necessity of the two terms. Diagonal-only sets μ=0\mu = 0 and keeps λI+Diag(qk)\lambda I + \mathrm{Diag}(q_k); rank-one-only sets qk=0q_k = 0 and keeps λI+μ,vkvk⊤\lambda I + \mu\\, v_k v_k^{\top}; everything else is Fed3DLoRA.

Table Z2. Fed3DLoRA minus variant, final round, mean over 3 seeds; accuracy points / choice NLL.

VariantCS8 / Qwen3-0.6BCS8 / Qwen3-1.7BMultiNLI / Qwen3-0.6BMultiNLI / Qwen3-1.7BDiagonal-only (μ=0)+1.22 / −0.0196+2.24 / −0.0346+0.24 / −0.0219+0.73 / −0.0310Rank-one-only (qk=0)+3.21 / −0.0514+6.60 / −0.1911+0.07 / −0.0045+6.76 / −0.1752\begin{array}{l|cccc} \text{Variant} & \text{CS8 / Qwen3-0.6B} & \text{CS8 / Qwen3-1.7B} & \text{MultiNLI / Qwen3-0.6B} & \text{MultiNLI / Qwen3-1.7B} \\\\ \hline \text{Diagonal-only (}\mu = 0\text{)} & \text{+1.22 / −0.0196} & \text{+2.24 / −0.0346} & \text{+0.24 / −0.0219} & \text{+0.73 / −0.0310} \\\\ \text{Rank-one-only (}q_{k} = 0\text{)} & \text{+3.21 / −0.0514} & \text{+6.60 / −0.1911} & \text{+0.07 / −0.0045} & \text{+6.76 / −0.1752} \end{array}

Both terms are needed, and in three of the four settings removing the rank-one term costs more. The one near-zero entry is exactly the case the reviewer identified from Table 21: on MultiNLI / Qwen3-0.6B the guard sets qk=0q_k = 0 for every client at rounds 5 and 10, so the full metric coincides with the rank-one-only metric for most of training there, and the end-to-end difference (+0.07) is what the fixed-input diagnostic predicted. Server time: the variants differ only in the C×C solve, a negligible part of the shared-subspace stages (Table R2 in our response to Reviewer NNmM; details under W2 below).

Threshold. (a) End to end, we trained Fed3DLoRA with the guard effectively removed (ε=10−30\varepsilon = 10^{-30} instead of the fp32 machine epsilon, nothing else changed). The final-round difference Fed3DLoRA minus guard-off is −0.03 / +0.17 / +0.09 / +0.29 accuracy points and −0.006 / −0.002 / −0.005 / −0.024 choice NLL on the four settings, within the seed-to-seed spread of Table 1 (SD 0.3–0.6). (b) On fixed inputs, we replayed the 27 recorded events (3 settings × 3 seeds × rounds 1/5/10, both projection families) in FP64 with the guard at ε=m⋅2−23\varepsilon = m \cdot 2^{-23} for m∈0,0.1,1,10,100,1000m \in \\{0, 0.1, 1, 10, 100, 1000\\} (m=1m = 1 is the submission; m=0m = 0 uses 10−3010^{-30}), with a relative guard (mean⁡(gk2)≤10−6⋅max⁡jmean⁡(gj2)\operatorname{mean}(g_k^2) \le 10^{-6} \cdot \max_j \operatorname{mean}(g_j^2)) and with all uploads scaled by α∈0.1,1,10\alpha \in \\{0.1, 1, 10\\}. Each configuration is scored one step ahead on the diagnostic set, applying the consensus to one projection family with the other family averaged, against product-space averaging of the same uploads at the same scale.

Table Z3. Fixed-input sweep of the guard: fraction of client–family observations with qk=0q_k = 0, and one-step accuracy / choice-NLL change relative to product-space averaging (medians over event families, then over seeds). Cell format: guard-active fraction, then Δaccuracy / ΔNLL.

\begin{array}{l|ccc} \text{m} & \text{CS8 / Qwen3-0.6B} & \text{CS8 / Qwen3-1.7B} & \text{MultiNLI / Qwen3-0.6B} \\\\ \hline \text{0} & \text{0\\%: +0.46 / −0.0008} & \text{0\\%: +0.39 / −0.0098} & \text{0\\%: +0.03 / −0.0010} \\\\ \text{0.1} & \text{2\\%: +0.46 / −0.0007} & \text{0\\%: +0.39 / −0.0098} & \text{37\\%: +0.03 / −0.0019} \\\\ \text{1 (paper)} & \text{32\\%: +0.20 / −0.0023} & \text{0\\%: +0.39 / −0.0098} & \text{67\\%: +0.10 / −0.0029} \\\\ \text{10} & \text{73\\%: +0.26 / −0.0078} & \text{5\\%: +0.39 / −0.0112} & \text{100\\%: +0.13 / −0.0017} \\\\ \text{100} & \text{100\\%: −0.91 / −0.0188} & \text{46\\%: +1.17 / −0.0373} & \text{100\\%: +0.13 / −0.0017} \\\\ \text{1000} & \text{100\\%: −0.91 / −0.0188} & \text{82\\%: +2.34 / −0.0845} & \text{100\\%: +0.13 / −0.0017} \end{array}

Within an order of magnitude of the submitted value (m from 0.1 to 10) the one-step effect relative to averaging changes by at most 0.26 points and keeps its sign in every setting; the relative guard never fires on these events and reproduces m = 0. The one-step outcome changes materially only when the guard is set two to three orders of magnitude above machine epsilon, where it removes the diagonal term for most or all clients. Update scale: since qkq_k and vkv_k are scale-free, the scale of the uploads enters the rule only through the guard. Scaling all uploads by 0.1 pushes the guard to fire for 46–100% of clients, scaling by 10 switches the guard off for every client, and the aggregated direction then coincides with the m = 0 case (identical relative change in all three settings), so the scale question reduces to the threshold question above. The recorded updates lie between 0.07× and 1840× the machine epsilon (Table 21), so the absolute guard does change which term is active across models and rounds, as Appendix A.3 states; what these measurements add is that this matters little: within an order of magnitude on either side of its value it moves the one-step effect by ≤ 0.26 points, and removing it altogether moves final accuracy by ≤ 0.29 points. We will add Table Z3 and the guard-off runs to Appendix E.4.

Q3 / W1. Accuracy–runtime trade-off against lightweight cross-layer weighting rules.

We trained four scalar rules under the same client protocol and write-back, each processing the two projection families separately (DkD_k is client kk's reference-corrected cross-layer update, wkw_k its weight, gkg_k its core, pp the core dimension):

  • xl_norm: ak∝wk∣Dk∣Fa_k \propto w_k \\|D_k\\|_F; weighted mean of the DkD_k, same rank-64 write-back as product-space averaging.
  • xl_sim: ak∝wk(1+cˉk)a_k \propto w_k (1 + \bar{c}_k), where cˉk\bar{c}_k is the mean cosine between vec,Dk\mathrm{vec}\\, D_k and the other clients' vec,Dj\mathrm{vec}\\, D_j; weighted mean of the DkD_k.
  • xl_sim_core: as xl_sim, but the cosines are computed on the shared cores gkg_k; the resulting scalar weights the whole DkD_k.
  • scalar_trace: the metric collapsed to one scalar per client, mk=tr⁡(Mk)/p=λ+mean⁡(qk)+μ∣vk∣2/pm_k = \operatorname{tr}(M_k)/p = \lambda + \operatorname{mean}(q_k) + \mu \\|v_k\\|^2/p; g⋆=∑kwkmkgk/∑kwkmkg^{\star} = \sum_k w_k m_k g_k \big/ \sum_k w_k m_k, with the reconstruction and write-back of Fed3DLoRA.

All four reduce to product-space averaging when the norms, similarities or traces are equal (checked in float64 to a relative error of 10⁻⁸).

Table Z4. Fed3DLoRA minus rule, final round, mean over 3 seeds; accuracy points / choice NLL.

\begin{array}{l|cccc} \text{Rule} & \text{CS8 / Qwen3-0.6B} & \text{CS8 / Qwen3-1.7B} & \text{MultiNLI / Qwen3-0.6B} & \text{MultiNLI / Qwen3-1.7B} \\\\ \hline \text{xl_norm} & \text{+0.39 / −0.0037} & \text{+1.22 / −0.1088} & \text{+0.31 / −0.0188} & \text{+0.71 / −0.0232} \\\\ \text{xl_sim} & \text{−0.22 / +0.0259} & \text{+0.09 / −0.0602} & \text{+0.47 / −0.0216} & \text{+0.58 / −0.0188} \\\\ \text{xl_sim_core} & \text{−0.35 / +0.0260} & \text{+0.35 / −0.0615} & \text{+0.49 / −0.0238} & \text{+0.60 / −0.0157} \\\\ \text{scalar_trace} & \text{+0.17 / +0.0139} & \text{+0.82 / −0.0754} & \text{+0.27 / −0.0227} & \text{+0.47 / −0.0187} \end{array}

Fed3DLoRA is better in 14 of the 16 accuracy comparisons and 13 of the 16 NLL comparisons. All exceptions are on CS8 / Qwen3-0.6B: there the two similarity-weighted rules are better on both metrics (by 0.22 and 0.35 points and 0.026 NLL), and scalar_trace has lower NLL (by 0.014) at lower accuracy. The norm-weighted rule, the closest to "importance = magnitude", is the weakest of the four in three of the four settings. On runtime: all four rules share the write-back; xl_norm and xl_sim need only norms and cosines of the updates, xl_sim_core and scalar_trace also need the bases and cores. The full metric therefore costs at most the shared-subspace stages of our GPU implementation, 0.13–0.47 s per round (Table R2), i.e. ≤ 0.3% of the summed client training time of a round. The trade-off is ≤ 0.5 s per round for +0.3 to +1.2 points over the norm rule in every setting and +0.1 to +0.6 points over the similarity rules in three of four settings, with one setting where the two similarity-weighted rules are slightly better. We will add the four rules to Appendix E and their definitions to Appendix B.5.

Response to Reviewer Zdmz (part 5 of 5)

W2, on the cost of the full procedure. The reviewer's reading of Appendix F is correct and the fault is ours: Tables 22–23 timed a dense fp32 implementation on four CPU threads. We have since implemented the same operator on the GPU (shared bases by blocked Householder QR, Gram operators and cores formed from the uploaded factors, the same rank-64 truncated SVD computed exactly through a QR of the low-rank factors and an SVD of the small core; the method, its hyperparameters and the client protocol are unchanged, and every snapshot agrees with the CPU implementation to a relative error of at most 10⁻⁴ on the aggregated update and on every client's download). One aggregation of both families now takes 1.26 / 1.36 / 0.91 / 1.78 s for Qwen3-0.6B / Qwen3-1.7B / Llama-3.2-1B / Llama-3.2-3B instead of 32.4 / 81.8 / 37.9 / 201.6 s; the stages specific to Fed3DLoRA take 0.13–0.47 s; the ratio to product-space averaging under the same write-back is 2.1–2.6× on absolute differences of 0.5–1.1 s. Relative to the logged client training time of the same rounds, the server step is 1.63% / 0.90% / 0.76% of the summed client time and 11.3% / 6.4% / 4.7% of the slowest client on CS8 / Qwen3-0.6B, CS8 / Qwen3-1.7B and MultiNLI / Qwen3-0.6B, against 24.9–42.0% and 71–298% in Table 22. Under one harness with six external baselines (Table R4 in our response to Reviewer NNmM), the whole server procedure of Fed3DLoRA takes 2.4 s per round against 1.0–3.8 s for the baselines, with communication equal to the lowest in the group; the full tables (R2–R4) are there and will replace Tables 22–23 in the revision.

Minor points. Paired per-seed differences: for the server-step control they are in Table 18; for the new 8B batch (Commonsense8 / Qwen3-8B-Base, seven external baselines, three seeds; Table R5 in our response to Reviewer NNmM) Fed3DLoRA is at or above every baseline in every seed, 19 strict wins and 2 exact ties in 21 paired comparisons, with a mean margin of 0.61 points over the strongest baseline. Task-level breakdown: we have it for the 8B batch (Table Z5 below; 96 examples per task, so one example is 1.04 points and a three-seed task mean carries a standard error of roughly 2.5 points). Fed3DLoRA has the highest mean on ARC-Easy, ARC-Challenge and HellaSwag, ties for the highest on OpenBookQA, is within 0.3–1.7 points of the best method on SocialIQA, BoolQ and PIQA, and is the lowest on WinoGrande (84.7 against 86.8–89.6). The mean gain at 8B therefore comes from the two ARC tasks and HellaSwag rather than from a uniform shift across tasks; we will add the table to Appendix D. For the runs of Tables 1–2 we did not store per-example predictions. Figures and tables: in the revision Figure 2 gets larger annotations and a legend outside the panels, Table 4 is split into its round-1 and rounds-5/10 halves, and Figure 3 gets explicit axis and panel labels.

Table Z5. Per-task final-round accuracy (%) on the 8B batch, mean over 3 seeds, 96 examples per task; best per task in bold. Every method is read at the same round for every task (no per-task round selection).

MethodARC-eARC-cBoolQHellaSwagOBQAPIQASIQAWinoGrandeFed3DLoRA88.5465.2885.7677.4348.2684.0369.1084.72Fed-PLoRA88.1961.4685.7675.0047.5784.7267.7187.85FSLoRA84.7264.5885.4277.0848.2683.3367.3686.81HetLoRA82.9962.5086.8174.6546.8885.7669.4487.85GLoRA84.3858.3387.1575.3543.0680.2168.7587.85FLoRA76.3954.8687.1575.0040.2885.0767.3689.58FedHera76.0454.5186.4675.6943.4081.2567.0188.54FlexLoRA78.4757.6485.4275.0039.2481.2564.5887.15\begin{array}{l|cccccccc} \text{Method} & \text{ARC-e} & \text{ARC-c} & \text{BoolQ} & \text{HellaSwag} & \text{OBQA} & \text{PIQA} & \text{SIQA} & \text{WinoGrande} \\\\ \hline \text{Fed3DLoRA} & \textbf{88.54} & \textbf{65.28} & \text{85.76} & \textbf{77.43} & \textbf{48.26} & \text{84.03} & \text{69.10} & \text{84.72} \\\\ \text{Fed-PLoRA} & \text{88.19} & \text{61.46} & \text{85.76} & \text{75.00} & \text{47.57} & \text{84.72} & \text{67.71} & \text{87.85} \\\\ \text{FSLoRA} & \text{84.72} & \text{64.58} & \text{85.42} & \text{77.08} & \textbf{48.26} & \text{83.33} & \text{67.36} & \text{86.81} \\\\ \text{HetLoRA} & \text{82.99} & \text{62.50} & \text{86.81} & \text{74.65} & \text{46.88} & \textbf{85.76} & \textbf{69.44} & \text{87.85} \\\\ \text{GLoRA} & \text{84.38} & \text{58.33} & \textbf{87.15} & \text{75.35} & \text{43.06} & \text{80.21} & \text{68.75} & \text{87.85} \\\\ \text{FLoRA} & \text{76.39} & \text{54.86} & \textbf{87.15} & \text{75.00} & \text{40.28} & \text{85.07} & \text{67.36} & \textbf{89.58} \\\\ \text{FedHera} & \text{76.04} & \text{54.51} & \text{86.46} & \text{75.69} & \text{43.40} & \text{81.25} & \text{67.01} & \text{88.54} \\\\ \text{FlexLoRA} & \text{78.47} & \text{57.64} & \text{85.42} & \text{75.00} & \text{39.24} & \text{81.25} & \text{64.58} & \text{87.15} \end{array}

We are grateful for a review that told us exactly which controls were missing. All of them are now run, and all of them will be in the revised PDF.

Reviewer_NNmMOpenReview ↗
Rating
3(+1)
Confidence
3(−1)
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a server-side aggregation method for federated LoRA with different client ranks. The server measures each client’s update from the adapter, builds a shared representation across layers, and combines the updates using client-specific metrics. The paper provides several properties of the aggregation rule and a conditional stationarity bound. Experiments show competitive accuracy and improvements over baselines.

Critical strengths
  1. The paper is structured very well, with a clear format that is easy to follow.
  2. The internal control comparisons are well selected to demonstrate the mechanism and motivation of the work.
  3. The experiment scope of this paper is broad. The paper covers 4 backbones, 40 clients, various client ranks, 9 baselines, and 3 seeds. The paper also reports the experimental setup in detail, which ensures the reproducibility of the work.
Critical weaknesses
  1. The format of the main context is good but overly dense. The theoretical derivation is also unreadable. It would improve the reviewer's experience if the authors could reformat the main context as well as the derivation.
  2. Overall, for five propositions, none of their insights are highlighted and clear. The significance of these propositions remains questionable.
  3. The empirical gain of the proposed method is not stable when compared with the baselines. The adopted model scale ends up at 3B, which is relatively small.
  4. The layer-wise control in Table 3 changes both the shared bases and the metric. It is unclear how much of the gain comes from sharing the bases and how much comes from combining information across layers.
Questions for authors
  1. Can the authors show how much of the gain comes from the shared bases? And can the authors keep the shared bases fixed and demonstrate whether or how much the metric combines information across layers?
  2. Why does the server computation overhead increase so much? Is this considered a limitation of the work?
  3. Can the authors show the comparison of the accuracy against total running time using the logged experiments? This would help assess whether the gains could justify the trade-off of the extra server computation.
  4. Does the advantage of the proposed method hold, degrade, or improve on a bigger model?
7 comments

Response to Reviewer NNmM (part 1 of 5)

We thank the reviewer for the careful reading, and for noting the structure, the internal controls and the breadth of the experiments. Before the individual answers, one correction that affects Q2 and Q3: the server times in Appendix F (Tables 22–23) measured a dense fp32 implementation running on 4 CPU threads (Appendix A.3, Table 14), written with correctness rather than speed as its priority. We have since implemented the same operator on the GPU. One aggregation now takes 0.9–1.8 s instead of 32–202 s, and the server step is 0.8–1.6% of the summed client training time of a round. All numbers below come from completed runs; no number in the main text changes, and the revision will replace Tables 22–23 and add the new tables.

Q1 / W4. How much of the gain comes from the shared bases, and does the metric combine information across layers when the bases are held fixed?

We ran the two controls that separate the two factors, on the four settings of Table 3 with three seeds each. This is a separate batch with its own seeds, so comparisons are paired within the batch (same data, ranks, participation sequence and initialisation per seed) but cannot be matched row by row with Table 3.

  • H1a, shared bases + layer-local metric. Keeps the shared Tucker bases of Fed3DLoRA, but the metric and the consensus of Eq. (8) are solved separately within each of the cL=16c_L = 16 retained layer-mode slices of the core. The aggregation decision for one slice therefore cannot use the energy profile or direction of any other slice, so nothing is coupled across the layer mode; this is the control the reviewer describes, expressed in the shared coordinates.
  • H1b, layer-wise bases + joint metric. Fits separate bases for every layer with the same total number of core coordinates as Fed3DLoRA (as the layer-wise control does), concatenates the per-layer cores, and runs the joint metric consensus over the concatenation.
  • LW, layer-wise bases + layer-wise metrics (the control of Table 3, re-run in this batch) completes the 2×2.

Table R1. Fed3DLoRA minus control, final round, mean over 3 seeds; accuracy points / choice NLL. Positive accuracy and negative NLL favour Fed3DLoRA. ControlCS8 / Qwen3-0.6BCS8 / Qwen3-1.7BMultiNLI / Qwen3-0.6BMultiNLI / Qwen3-1.7BH1a shared bases, layer-local metric+1.30 / −0.0172+2.86 / −0.1153+1.31 / −0.0161+0.62 / −0.0282H1b layer-wise bases, joint metric+0.17 / −0.0004+5.43 / −0.1672+3.40 / −0.0461+0.67 / −0.0239LW layer-wise bases and metrics+0.30 / −0.0083+3.56 / −0.1439+0.11 / −0.0188+0.84 / −0.0218\begin{array}{l|cccc} \text{Control} & \text{CS8 / Qwen3-0.6B} & \text{CS8 / Qwen3-1.7B} & \text{MultiNLI / Qwen3-0.6B} & \text{MultiNLI / Qwen3-1.7B} \\\\ \hline \text{H1a shared bases, layer-local metric} & \text{+1.30 / −0.0172} & \text{+2.86 / −0.1153} & \text{+1.31 / −0.0161} & \text{+0.62 / −0.0282} \\\\ \text{H1b layer-wise bases, joint metric} & \text{+0.17 / −0.0004} & \text{+5.43 / −0.1672} & \text{+3.40 / −0.0461} & \text{+0.67 / −0.0239} \\\\ \text{LW layer-wise bases and metrics} & \text{+0.30 / −0.0083} & \text{+3.56 / −0.1439} & \text{+0.11 / −0.0188} & \text{+0.84 / −0.0218} \end{array} To the two parts of the question: (i) with the shared bases held fixed, letting the metric couple layers is worth +0.62 to +2.86 points over the layer-local metric (H1a), with lower choice NLL in every setting; (ii) with the joint metric held fixed, the shared bases are worth +0.17 to +5.43 points over layer-wise bases (H1b), again with lower NLL in every setting. Neither factor alone recovers Fed3DLoRA, and the two contributions are not additive: the H1a and H1b gaps do not sum to the LW gap, and in three of the four settings one of the two single-factor designs (H1a or H1b) is worse than the consistent layer-wise control (LW). This is consistent with the construction. The metric compares coordinates across clients, so cross-layer coupling is only meaningful when the layers share coordinates, and shared coordinates only pay off when the consensus uses them jointly. We will add Table R1 next to Table 3 and rewrite the sentence of §4.3 that currently says the comparison does not isolate the two factors. A side note on the storage line of §4.3: recomputed from the actual q/v shapes and the mode ranks returned by the aggregation, the shared bases of both families take 2.5 / 3.5 / 3.3 / 5.0 MiB for Qwen3-0.6B / Qwen3-1.7B / Llama-3.2-1B / Llama-3.2-3B, against 58 / 74 / 52 / 104 MiB for layer-wise bases with the same number of core coordinates (16–23×). The Qwen values coincide with Appendix F.3 because the design is unchanged; the Llama values are new.

Response to Reviewer NNmM (part 2 of 5)

Q2. Why does the server computation overhead increase so much? Is this a limitation?

The reviewer's reading of Appendix F is correct, and the fault is ours: the size of the overhead came from how we measured it, not from the method, and Appendix F should have said so. Tables 22–23 timed the dense fp32 CPU implementation of Appendix A.3 (Xeon Gold 5318Y, 4 threads; Table 14). Every stage ran on four CPU threads, including the L rank-64 truncated SVDs of the write-back, which product-space averaging under the same write-back pays as well. The work specific to Fed3DLoRA, namely the three Gram operators, three eigendecompositions of sizes L, m and n, C projections and one C×C solve (§3.5), was carried out on dense m×n updates: forming the Gram operators this way costs on the order of L⋅m⋅n⋅(m+n)L \cdot m \cdot n \cdot (m+n) operations per client and family, and this dominated the CPU times of Tables 22–23.

We have implemented the same operator on the GPU: the shared bases are obtained by blocked Householder QR, the Gram operators and cores are formed directly from the uploaded factors instead of from dense updates, and the write-back computes the same rank-64 truncated SVD exactly through a QR factorisation of the low-rank factors followed by the SVD of the small core, with the small SVDs on a CPU worker thread overlapped with GPU work. The method is unchanged: the same mode ranks (16, 128, 128), the same λ\lambda, μ\mu, cc and guard, the same complement-preserving reconstruction and write-back, and no additional client computation or communication. Every snapshot reported below passed all numerical gates of our pipeline before it was timed: the aggregated global update and every client's download packet agree with the CPU implementation of the submission to a relative error of at most 10⁻⁴.

Table R2 (replaces Table 23). One aggregation of both families, median of 5 repetitions. Timed from decoded uploads and reference state on the CPU to the aggregated result on the CPU, including index planning, host–device copies, tail reconstruction and the aggregation diagnostics; excluding download packaging, network, disk, client training and evaluation. The product-space mean uses the same exact QR/SVD write-back. The write-back share and the subspace-stage times come from three additional segmented runs (per-run ratio of write-back wall-clock to full aggregation wall-clock, and the sum of the Gram, eigenbasis, core-projection and metric/consensus stages; medians over runs), without double-counting work overlapped between CPU and GPU.

\begin{array}{l|ccccc} \text{Model} & \text{Fed3DLoRA (s)} & \text{was, Table 23} & \begin{array}{c}\text{QR/SVD}\\\\\text{write-back share}\end{array} & \begin{array}{c}\text{shared-subspace}\\\\\text{stages (s)}\end{array} & \begin{array}{c}\text{product-space mean,}\\\\\text{same write-back (s)}\end{array} \\\\ \hline \text{Qwen3-0.6B-Base} & \text{1.258} & \text{32.4} & \text{74.0\\%} & \text{0.134} & \text{0.591} \\\\ \text{Qwen3-1.7B} & \text{1.363} & \text{81.8} & \text{67.4\\%} & \text{0.197} & \text{0.619} \\\\ \text{Llama-3.2-1B †} & \text{0.907} & \text{37.9} & \text{64.0\\%} & \text{0.189} & \text{0.402} \\\\ \text{Llama-3.2-3B †} & \text{1.784} & \text{201.6} & \text{52.6\\%} & \text{0.471} & \text{0.697} \end{array}

† No client uploads of the Llama runs were stored, so these rows time the server on the real checkpoints with controlled updates applied to the active prefix. They are real executions with the real shapes (server time does not depend on the update values), not replays of training updates, and they carry no accuracy claim. The Qwen rows replay recorded uploads.

The stages specific to Fed3DLoRA (Gram operators, eigenbases, core projections, metric and consensus) take 0.13–0.47 s per round; the write-back now accounts for 53–74% of the total because these stages shrank by more than the write-back did. The difference to the product-space mean under the same write-back is 0.5–1.1 s per round; the ratio quoted in Appendix F (2.6–3.3×) is now 2.1–2.6×, but on absolute numbers that are two orders of magnitude smaller.

Response to Reviewer NNmM (part 3,4 of 5, Q2 continued)

Table R3 (replaces Table 22). The same aggregation relative to the logged client training time of the replayed round (CS8 / Qwen3-0.6B: seed 47, round 0; the other two: seed 11, round 0). Denominators are the clients' logged elapsed_seconds of that round, so these are replay estimates on one round per setting, not medians over the 30 logged rounds of Table 22.

\begin{array}{l|cccccc} \text{Setting} & \text{server (s)} & \text{was} & \begin{array}{c}\text{of summed}\\\\\text{client time}\end{array} & \text{was} & \begin{array}{c}\text{of slowest}\\\\\text{client}\end{array} & \text{was} \\\\ \hline \text{CS8 / Qwen3-0.6B} & \text{1.258} & \text{19.27} & \text{1.63\\%} & \text{24.9\\%} & \text{11.3\\%} & \text{161\\%} \\\\ \text{CS8 / Qwen3-1.7B} & \text{1.363} & \text{62.48} & \text{0.90\\%} & \text{42.0\\%} & \text{6.4\\%} & \text{298\\%} \\\\ \text{MultiNLI / Qwen3-0.6B} & \text{1.315} & \text{19.38} & \text{0.76\\%} & \text{10.9\\%} & \text{4.7\\%} & \text{71\\%} \end{array}

Is it a limitation? With this implementation we do not think the server time is a practical one: it is about 1% of the client compute of a round and 5–11% of the slowest client, so under synchronous rounds the aggregation adds about 1.3 s to rounds of 11–28 s (the whole server procedure, including download packaging, is timed under Q3, Table R4). What remains is that these times rely on a GPU at the server: on four CPU threads the same operator is the 32–202 s of the submission. We will state this, together with the server-side memory of the operator, explicitly in the limitations. In the revision we will (a) replace Tables 22–23 and the text of F.2 by Tables R2–R3, (b) correct Appendix A.3 and Table 14, which currently say the server runs on the CPU, keeping the CPU implementation as the reference for the numerical checks of Appendix C.4, and (c) replace the "2.6–3.3× slower" sentence by absolute times.

Q3. Accuracy against total running time using the logged experiments.

We agree this is the right test, and it can be answered from logged and measured quantities without new training. Every method of a batch trains the same client protocol (Table 9), so the client time of a round is nominally the same for every method, and the server time is now a small addition to it. Table R4 times the complete server procedure of all seven methods under one harness on the same saved round, with the per-round communication, next to the final accuracy of Table 1.

Table R4. Whole-server time per round for CS8 / Qwen3-1.7B (one saved round, 8 clients), all seven methods under one harness on the same A100 server: from decoded uploads to all eight serialised download packets, excluding network, disk, client training and evaluation; median of five repetitions with the observed range. Baselines run on the same saved uploads, adapted to their formats, not retrained. Communication: serialised message bytes per round. Accuracy: Table 1.

\begin{array}{l|cccccc} \text{Method} & \begin{array}{c}\text{server procedure,}\\\\\text{median (s)}\end{array} & \begin{array}{c}\text{range of}\\\\\text{5 runs (s)}\end{array} & \text{upload (MiB)} & \text{download (MiB)} & \text{both (MiB)} & \begin{array}{c}\text{final accuracy,}\\\\\text{Table 1 (\\%)}\end{array} \\\\ \hline \text{HetLoRA} & \text{0.987} & \text{0.45–1.21} & \text{184.0} & \text{183.9} & \text{367.9} & \text{68.06} \\\\ \text{FlexLoRA} & \text{1.251} & \text{0.75–1.30} & \text{184.0} & \text{183.9} & \text{367.9} & \text{67.14} \\\\ \text{Fed-PLoRA} & \text{1.544} & \text{0.68–1.57} & \text{184.1} & \text{392.2} & \text{576.3} & \text{66.75} \\\\ \text{FSLoRA} & \text{2.193} & \text{0.67–2.50} & \text{184.1} & \text{184.0} & \text{368.1} & \text{68.66} \\\\ \text{FLoRA (native stacked broadcast)} & \text{2.228} & \text{1.59–2.28} & \text{184.0} & \text{1470.1} & \text{1654.2} & \text{67.19} \\\\ \text{Fed3DLoRA} & \text{2.383} & \text{1.87–2.76} & \text{184.0} & \text{183.9} & \text{367.9} & \text{68.79} \\\\ \text{FedHera} & \text{3.772} & \text{3.18–5.53} & \text{318.8} & \text{318.9} & \text{637.7} & \text{67.27} \end{array} All seven server procedures take between 1.0 and 3.8 s per round. FThe additional server computation is real and we do not claim otherwise; what matters is its size: 1.6% of the summed client training time of a round (≈ 151 s) and 11% of the slowest client's. On communication Fed3DLoRA is in the lowest tier because clients exchange only their rank-rkr_k factors and the bases and cores never leave the server. With the CPU numbers of the submission a round would have been almost 4× longer than the fastest baseline's; that is the trade-off the reviewer rightly questioned, and it no longer exists. We will add Table R4 to Appendix F.

Comments on the paper

1 comment

Summary of the rebuttal and revision

We thank the Area Chair and the three reviewers. The revised PDF is uploaded (changes in blue, main text within 10 pages; the abstract field matches the PDF).

New experiments added during the discussion (same protocol, paired within each batch):

  • Shared bases vs. cross-layer metric: controls changing one factor at a time; neither recovers Fed3DLoRA (Table 3, lower block).
  • Metric design: diagonal-only, rank-one-only and four scalar weighting rules, all end to end (Appendix E.6, Tables 26–27); small-energy guard swept on fixed inputs and removed end to end (Appendix E.4, Table 24).
  • Shared coordinates: per-layer energy capture, median 32–73% per layer, 26–167× a random subspace (Appendix E.4, Table 25); Proposition 6 on sign and rotation invariances with measured perturbations (Appendix C.2, E.7, Table 28).
  • Server cost: GPU timings replacing the CPU tables, 0.9–1.8 s per aggregation; seven-method harness on one saved round, 2.4 s per round, 1.6% of client training time, communication in the lowest tier (§3.5, Appendix F.2, Tables 29–31).
  • Scaling: Qwen3-8B-Base against seven baselines, highest mean accuracy, at or above every baseline in every seed (§4.2, Appendix D.5, Tables 20–21).

By reviewer. NNmM: ablation, cost, scaling and limitations above; the follow-up questions (Tucker slices vs. layers, time to a given accuracy, FLoRG) are answered in the thread. Zdmz: presentation of §3, metric variants, guard sensitivity and the 8B per-task breakdown. jQcq: shared coordinates, the metric and its link to the objective, the scope of Theorem 1 and basis dependence; the two remaining questions are answered in our second reply with the shuffled-metric, step-size and one-step controls (Tables 3, 5, 24).

We are grateful for the suggestions, which have made the paper more complete.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.