- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper proposes Fed3DLoRA, a server-side aggregation method for federated LoRA with heterogeneous client ranks and data distributions. The method first corrects each client update against the adapter prefix it actually received, then stacks updates from the same projection family across Transformer layers into a third-order tensor. Shared layer, input, and output bases are used to obtain compact client cores, and a client-specific metric based on update energy and direction is used to compute a cross-layer consensus. The consensus is applied only inside the shared subspace, while the product-space mean is kept in the orthogonal complement, followed by a rank-constrained spectral write-back for clients with different ranks. The paper proves gauge invariance, zero-update consistency, reduction to averaging under identical metrics, a bounded deviation from averaging, and a conditional stationarity result. Experiments on Commonsense8 and MultiNLI with Qwen3 and Llama backbones show that Fed3DLoRA is competitive with existing heterogeneous federated fine-tuning methods and generally improves over layer-wise and product-space averaging controls.
Critical strengths
Strength 1: Fed3DLoRA jointly models client updates across layers instead of aggregating each layer independently.
Strength 2: The reference-correction step cleanly handles clients with different LoRA ranks.
Strength 3: The method keeps client-side computation and communication unchanged while improving server-side aggregation.
Critical weaknesses
Weakness 1: Fed3DLoRA assumes that updates from different Transformer layers can be meaningfully placed in one shared tensor space. The paper uses common input and output bases across layers, yet it is not clear why update directions from different depths should be comparable in the same coordinates.
Weakness 2: The client metric is built from the update magnitude and direction. This makes the aggregation rule depend heavily on how large a client update is. A large update may carry useful information, or it may come from client drift or noisy local training. The paper does not show why these quantities are reliable signals for client importance, and the chosen metric form is not directly connected to the federated objective.
Weakness 3: The convergence analysis mainly shows that the method remains controlled when its update stays close to product-space averaging. It does not show that the proposed cross-layer consensus gives a better estimate of the global update, reduces client drift, or leads to a better optimization direction. This leaves a gap between the motivation of the method and what the theory actually proves.
Weakness 4: The aggregation rule depends on the specific basis used inside the shared subspace. The diagonal metric is not invariant to rotations of this basis, so two different orthonormal bases spanning the same subspace can produce different client metrics and different aggregated updates. This makes the aggregation sensitive to the basis choice and gives no guarantee that the result is determined by the shared subspace itself.
Questions for authors
Question 1: Why should updates from different Transformer layers share the same input and output bases, and under what conditions is this cross-layer representation expected to hold?
Question 2: Why are update magnitude and direction used to define client importance in the metric? How does this rule distinguish useful task information from client drift or noisy local updates?
Question 3: Can the authors show that the proposed consensus gives a better estimate of the global update or reduces client drift, instead of only bounding its deviation from product-space averaging?
Question 4: Since the diagonal metric is not invariant to rotations within the shared subspace, how can the method guarantee stable aggregation when different equivalent bases can produce different aggregated updates?
7 comments
Response to Reviewer jQcq (part 2 of 4)
Q2 / W2. Why magnitude and direction define the metric; how the rule treats drift or noise; the link to the federated objective.
First, a correction of the premise. The metric does not depend on how large a client's update is: is the relative energy profile (squared coordinates divided by their mean, clipped at ) and is a unit vector, so is invariant to for every . A client's magnitude enters the consensus only through itself in , exactly as it enters averaging. What the metric encodes is where (which coordinates) and along which direction a client's update lies, not how big it is. Two controls confirm that magnitude is not what the gain comes from: the norm-weighted rule , the literal "importance = size" rule trained under the same protocol, is worse than Fed3DLoRA in all four settings (by 0.31–1.22 points, with higher choice NLL in each); and rescaling the Fed3DLoRA update to the norm of the averaging update in every round (Fed3DLoRA-NM, Table 5) keeps the accuracy gain over averaging and lowers choice NLL further in every seed.
Link to the objective. Eq. (8) is the minimiser of . Read as an objective, this replaces each client's local loss by a quadratic model around that client's own update, with curvature , and takes the minimiser of the weighted sum of these models. is a damped, normalised diagonal-plus-rank-one proxy of the client's empirical Fisher information in the shared coordinates (the outer product of the client's own update direction, and its diagonal), so the rule is Fisher-weighted merging (Matena & Raffel, 2022; cited in §5), applied per round in the cross-layer coordinates and with the complement averaged. It is a proxy: it is built from the client's aggregated update rather than from per-example gradients, and we do not claim it equals the Fisher. We will add this derivation to §3.3, which currently motivates the two terms only geometrically.
Drift and noise. We do not claim that the metric identifies drift, and we will not claim it in the revision. What the rule does with a client that disagrees with the others is characterised rather than assumed: Prop. 5 gives the closed form of the rank-one consensus, in which a direction shared by no other client is scaled by , a bounded factor; Prop. 4 bounds the total deviation from the mean by ; and the guard removes the diagonal term for updates at machine-epsilon scale. That the quantities the metric uses are informative rather than arbitrary is tested by the shuffled-metric control (Table 3), which keeps every metric's spectrum but breaks its alignment with the client's update and loses 0.1–1.8 points, and by the fixed-input evidence under Q3.
Response to Reviewer jQcq (part 3 of 4)
Q3 / W3. Does the consensus give a better estimate of the global update, or only a bounded deviation?
We agree on what Theorem 1 shows and does not show. It is a stability result: the iterates stay within a bounded distance of product-space local SGD, and the bound reduces to that of averaging when the client metrics coincide. It is not an improvement result, and the revision will say so in those words in §3.5 and in the conclusion. Nor do we claim that the consensus is a closer estimate of the full-participation update than the cohort mean is: the cohort mean is the natural estimator of that quantity, and the consensus departs from it inside the shared subspace by design (Props. 3–4). What we claim is that this departure improves the optimisation outcome, and that claim is empirical. We have three kinds of evidence; the first is on identical inputs, where no training-trajectory confound exists. (a) Fixed-input one-step evaluation. On the 27 recorded events (3 settings × 3 seeds × rounds 1/5/10) we apply the consensus and the product-space mean of the same uploads to the same global adapter and score the result on the held-out diagnostic set, one projection family at a time with the other averaged. The consensus gives higher accuracy and lower choice NLL than the mean in every setting: medians over events of +0.20 / −0.0023 on CS8 / Qwen3-0.6B, +0.39 / −0.0098 on CS8 / Qwen3-1.7B and +0.10 / −0.0029 on MultiNLI / Qwen3-0.6B. Per event–family, one-step accuracy is above averaging in 37 of the 54 cases, equal in 3 and below in 14, and the count does not fall over training (11, 13 and 13 of 18 at rounds 1, 5 and 10). (b) Scalar steps cannot reproduce it. No global step on the mean, including the FedExP step, reaches Fed3DLoRA on CS8 / Qwen3-1.7B, which is above the mean in every seed (Table 5, Appendix D.3), and the norm-matched variant shows the gain is not a scale effect. (c) Every simplification of the rule is worse end to end. Layer-local coupling, diagonal-only and rank-one-only metrics, and four scalar-weighted rules (norm, similarity, core similarity, trace), each trained under the same protocol on the four settings of Table 3 with three seeds, are behind Fed3DLoRA in 14 of 16 accuracy comparisons for the scalar rules and in all 12 for the others (full tables in our responses to Reviewers NNmM and Zdmz). We will state the empirical status of the claim precisely in the revision.
Response to Reviewer jQcq (part 1 of 4)
We thank the reviewer for a precise review. The four observations are mathematically correct, and the paper does not claim otherwise; where it was not explicit enough about them, it should have been. Our answers therefore do not dispute the observations. For each, we state exactly what the construction guarantees, quantify on recorded uploads and in end-to-end runs how much the property matters, and name the additions we will make to the text.
Q1 / W1. Why should updates from different layers share input and output bases, and when is this expected to hold?
Three parts. (i) Structure. The input mode of a projection family is the residual stream: every layer's query and value projections read the same hidden-state coordinates, so an input basis UI shared across layers is a basis of a space the layers do share. The output mode (per-layer head coordinates) and the layer mode are not shared by construction. There, the shared bases are not an assumption about the updates but a fitted coordinate system: the leading eigenvectors of the cohort's energy-weighted Gram operators (Eq. 4), i.e. the coordinates that capture the most update energy for the prescribed dimension, re-estimated in every round. We do not assume that updates at different depths are similar; we fit the cohort's common coordinates and measure what they capture. (ii) What happens when the representation is poor. The consensus acts only inside the shared subspace; the orthogonal complement is aggregated by the product-space mean (Prop. 3), and the deviation from the mean is bounded (Prop. 4). A shared subspace that captures little energy therefore makes Fed3DLoRA approach product-space averaging; it cannot move it away from averaging by more than the bound. (iii) When it holds, observably. The operative condition is the captured energy fraction, which the server computes in every round: with (cL,cO,cI)=(16,128,128) the shared subspace retains medians of 32–65% of the cohort's weighted update energy across settings and rounds (Appendix E.4), so the consensus acts on roughly half of the update and averaging on the rest. That fraction is not carried by a few dominant layers. Measured layer by layer on the 36 recorded events of all four settings (72 event–families, 2,016 layer records; output and input bases only, no layer-mode basis), the cohort-fitted bases capture a median of 32–73% of each layer's update energy, 26–167 times what a random subspace of the same dimensions captures; in every event at least 20 of the 28 layers, and typically 26–28, exceed ten times the random level, and the layer with the lowest capture in an event still lies several times above it (3–25% at rounds 5 and 10; in the setting we examined in detail these are the last two layers, holding under 0.5% of the family's update energy). Every layer therefore has a substantial component in the shared coordinates, and what falls outside them is averaged (Prop. 3). Two further measurements indicate that the shared coordinates are informative rather than arbitrary. Cores of clients with the same task-pair profile are closer than cores of clients with different profiles in all 36 recorded event–family combinations (Figure 6). And a new control that applies the joint metric on layer-wise bases (separate bases per layer with the same total number of core coordinates, cores concatenated) is worse than Fed3DLoRA by 0.17 / 5.43 / 3.40 / 0.67 accuracy points on CS8 / Qwen3-0.6B, CS8 / Qwen3-1.7B, MultiNLI / Qwen3-0.6B and MultiNLI / Qwen3-1.7B (three seeds each, paired; lower choice NLL for Fed3DLoRA in every setting), whereas the consistent layer-wise control is behind by only 0.30 / 3.56 / 0.11 / 0.84: a joint metric is useful only in shared coordinates. In the revision, §3.2 will state (i)–(iii) explicitly and name the energy fraction as the condition under which the cross-layer representation is doing work (with the per-layer capture table in Appendix E.4), and the abstract and introduction will present the shared residual stream as the reason the layers can be given common coordinates, not as a claim that their updates are similar.