acceptodds
Under review as a conference paper at ICLR 2027

FedSALT: Subspace-Drift-Guided Adaptive Local Training for Federated LoRA Fine-Tuning

Abstract

Federated low-rank adaptation (LoRA) enables efficient fine-tuning of large language models (LLMs) over decentralized data by communicating only compact low-rank adapters. However, heterogeneous client tasks naturally produce task-specific LoRA updates, making it difficult to learn a single global model through aggregation. We observe an empirical asymmetry in this heterogeneity: task differentiation is substantially more pronounced in the output-side geometry of LoRA updates than in the input-side geometry. Motivated by this observation, we propose FedSALT, a geometry-aware closed-loop optimization framework that maintains a momentum-based consensus subspace, measures client-module disagreement using a factorization-invariant output-subspace score computed directly from the realized LoRA update, and converts the score into adaptive proximal regularization for subsequent local optimization. Unlike existing methods that apply uniform regularization or intervene only during aggregation, FedSALT selectively guides drifting modules toward a momentum-lookahead consensus anchor while retaining standard factor-wise FedAvg aggregation. Experiments on three heterogeneous federated LLM benchmarks show that FedSALT consistently achieves the best overall accuracy among representative baselines. Mechanism studies further validate the motivating output-side asymmetry and demonstrate the effectiveness of adaptive geometry-aware control.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Federated learning, Low-rank adaptation, Large language models, Client heterogeneity, Subspace geometry, Adaptive regularization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_1LiaOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary
Paper Summary
  1. FedSALT addresses heterogeneous federated LoRA training by measuring how client updates diverge from a historical consensus output subspace.
  2. It converts factorization-invariant alignment scores into adaptive client- and module-specific proximal strengths, guiding subsequent local updates toward a momentum-based reference.
  3. Experiments on three heterogeneous benchmarks report improved average accuracy, supported by component ablations and an idealized convergence analysis.
Critical strengths
Critical Strengths
  1. Clear geometric motivation. The paper identifies stronger task differentiation in output-side than input-side update geometry and uses this observation to guide selective local regularization.
  2. Principled drift measurement. The alignment score operates on realized adapter updates, making it invariant to equivalent LoRA factorizations. Computation remains in low-rank form without constructing dense updates.
  3. Informative mechanism ablations. Input-side sensing, matched-mean uniform control, permuted assignments, and zero-centered targets help distinguish the contributions of geometry-aware allocation and momentum guidance.
Critical weaknesses
Critical Weaknesses
  1. The theory does not cover the implemented algorithm. The analysis assumes full participation, optimization directly in realized-adapter space, and exact product-space aggregation. The implementation instead trains LoRA factors and uses factor-wise FedAvg, leaving both factorization geometry and aggregation mismatch outside the guarantee.
  2. Empirical gains are modest and narrowly validated. Evaluation uses only Llama-3.2-1B, rank 16, and eight-client federations. Improvements over the strongest baseline are approximately 0.3, 0.5, and 0.3 percentage points across the three benchmarks; broader settings and paired statistical comparisons would strengthen the evidence.
  3. Practical trade-offs need further evaluation. Table 4 reports 1.50× communication and 6.3% runtime overhead relative to FedAvg. Sensitivity to controller parameters and reference ranks, together with accuracy comparisons under matched communication or time budgets, is needed to establish whether the gains justify the additional complexity.
Questions for authors

Please refer to weakness.

4 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer 1Lia

We thank the reviewer for highlighting the clear geometric motivation, the principled drift measurement, and the informative mechanism ablations.

Tables report client-average test accuracy (%) as mean ± standard deviation over three seeds.

W1

Standard analytical setting. Full participation and exact averaging of client updates are the standard setting of federated optimization analyses. For example, Li et al. [1, Theorem 1] analyze FedAvg under full participation with weighted model averaging, and Stich [2, Algorithm 1] analyzes local SGD with periodic exact averaging.

Analysis in the space where FedSALT acts. Both the factorization-invariant alignment score and the adaptive proximal objective are defined on the realized LoRA update, so the realized-adapter space is exactly where FedSALT senses and controls drift. In this space, Proposition 1 establishes that stronger proximal control contracts the leading off-subspace component of local drift, and Theorem 1 gives a nonconvex stationarity bound that explicitly quantifies selection bias, client heterogeneity, and momentum-anchor effects.

Factor-wise aggregation. FedSALT changes only the local objective and keeps standard factor-wise FedAvg aggregation, so the analysis isolates exactly the component that FedSALT contributes. We will add this discussion and the references to Section 4.

[1] Li et al., On the Convergence of FedAvg on Non-IID Data, ICLR 2020.

[2] Stich, Local SGD Converges Fast and Communicates Little, ICLR 2019.

W2

Paired comparison. The table below reports a paired comparison on MC-8 over three seeds. FedSALT is more accurate than every baseline on every seed, and the improvement is statistically significant against every baseline.

BaselineSeeds where FedSALT is higherOne-sided paired t-test p-value
FedRot-LoRA3/30.030
FedProx3/30.012
SCAFFOLD3/30.016
FedEx-LoRA3/31.1 × 10⁻⁴
FedAvg3/30.013
FedSA-LoRA3/30.0017
FFA-LoRA3/30.0026

Additional backbones. The table below reports MC-8 with Qwen2.5-1.5B and Llama-3.2-3B under the protocol of Section 5. FedSALT remains the most accurate method on both backbones.

MethodQwen2.5-1.5BLlama-3.2-3B
FedSALT68.76 ± 0.0867.48 ± 0.11
FedRot-LoRA68.44 ± 0.2167.38 ± 0.22
FedProx68.62 ± 0.3167.01 ± 0.13
SCAFFOLD68.44 ± 0.1567.10 ± 0.18
FedEx-LoRA68.53 ± 0.5067.38 ± 0.29
FedAvg68.57 ± 0.4267.34 ± 0.16
FedSA-LoRA67.94 ± 0.0667.18 ± 0.02
FFA-LoRA68.26 ± 0.2366.02 ± 0.04

More clients. MC-12 extends MC-8 to 12 clients. FedSALT remains the most accurate method with more clients.

MethodMC-12
FedSALT58.65 ± 0.08
FedRot-LoRA58.17 ± 0.17
FedProx58.60 ± 0.34
SCAFFOLD58.38 ± 0.33
FedEx-LoRA58.30 ± 0.19
FedAvg58.23 ± 0.12
FedSA-LoRA57.85 ± 0.16
FFA-LoRA57.07 ± 0.44

We will include the results with the additional backbones and with MC-12 in the revision.

W3

Matched time budget. We match wall-clock time on MC-12 by limiting FedSALT to the time FedAvg needs for its full training, computed from the per-round runtime in Table 4, while every baseline keeps its full training. Within this budget, FedSALT reaches the same accuracy as in the MC-12 table under W2 and remains the most accurate method.

Communication. FedSALT leaves the uplink unchanged, and its additional cost is a single downlink broadcast of one low-rank reference shared by all clients. Its total communication remains below SCAFFOLD, and its runtime overhead is smaller than that of FedProx, FedRot-LoRA, SCAFFOLD, and FedSA-LoRA (Table 4).

Sensitivity. The table below changes one controller parameter or the reference rank at a time on MC-8. FedSALT is robust to its controller parameters and reference rank.

SettingAccuracy (%)
Default (Table 3)58.52 ± 0.14
μmax⁡=0.075\mu_{\max}=0.07558.61 ± 0.02
μmax⁡=0.15\mu_{\max}=0.1558.29 ± 0.34
γ=0.95\gamma=0.9558.59 ± 0.32
rref=12r_{\mathrm{ref}}=1258.19 ± 0.27
AuthorsOpenReview ↗

Gentle Follow-up for Reviewer 1Lia on Our Rebuttal

Dear Reviewer 1Lia,

Thank you again for your review. As the discussion period closes today, we would like to briefly summarize our response.

  • Theory. Full participation and exact averaging are the standard setting of federated optimization analyses (e.g., Li et al., ICLR 2020; Stich, ICLR 2019). Our analysis is carried out in the realized-adapter space, where FedSALT senses and controls drift, and because FedSALT changes only the local objective while keeping factor-wise FedAvg, the analysis isolates exactly its contribution.
  • Strength of the gains. FedSALT is more accurate than every baseline on every seed, with statistically significant improvements, and remains the most accurate method with Qwen2.5-1.5B, Llama-3.2-3B, and 12 clients.
  • Trade-offs. Under a matched wall-clock budget, FedSALT remains the most accurate method. It leaves the uplink unchanged, its total communication remains below SCAFFOLD, and it is robust to its controller parameters and reference rank.

We sincerely appreciate your time and would be grateful for any feedback on whether our response addresses your concerns. If it does, we would be grateful if you would kindly consider updating your rating.

Thank you for your response. I will keep my score.

We thank the reviewer for reading our response.

We will incorporate the theoretical discussion, the paired comparison, the results with additional backbones and 12 clients, the matched time budget comparison, and the sensitivity analysis into the revised paper.

Reviewer_2NukOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes FedSALT to address federated LoRA fine-tuning. It uses an output subspace score for each client and module in the next training round to reduce the drift when clients have heterogeneous data. Experiments in three federated settings show improvements in the average achieved accuracy over the strongest baselines. The paper also provides theoretical analysis and ablation studies.

Critical strengths
  1. The problem the authors analyze is valid and useful. In federated LoRA, adjusting regularization can control local updates under different client data.
  2. The method updates the local training in the next round. This online control provides robustness along the training process.
  3. The method achieves the highest average accuracy in the experiments, showing its benefits. The results are over three random seeds and include uniform regularization with the same average strength and randomly assigned strengths, which are sound.
Critical weaknesses
  1. The writing is sometimes hard to follow and needs extra improvement. The same motivation is repeated in Sections 1, 2.1, 3, and 3.5, while the order of updates is not consistent and spread across several sections. The structure of this paper needs to be optimized as well. In the current version, it takes effort to understand what is calculated in the current round, what is used in the next round, and what the relationship and connection are between them. The paper should also clearly define the subspace similarity in Figure 1 and explain how the measurements are averaged.
  2. Some writing grammar exists. For example, there are multiple "Eq. equation" in the paper.
  3. The novelty and motivation are incremental. Earlier work has already studied the different roles of the two LoRA factors. The projection score, moving average, and momentum also build on known ideas. The main new part is using the output subspace score to adjust regularization for each client and module. However, the paper does not clearly show whether this score adds more benefits when compared with using simpler signals.
  4. In the comparison between the input side and output side, it is unclear whether the average regularization strength is matched. The reported off-subspace energy is a ratio. This ratio can fall because the update inside the reference subspace becomes larger, even if the update outside it does not become smaller. The current results therefore do not fully support the explanation of how the method reduces drift.
  5. The experiment scope is not enough. The paper mainly uses Llama3.2-1B, which is small. Besides, experiments use rank 8 and only 8 clients. It is unclear if the proposed method holds its advantage when the scale of the model, number of clients, and rank change. In addition, the reported average gain over the baseline is marginal, while its communication overhead is much larger than the baseline. The trade-off needs to be well justified.
Questions for authors
  1. What are the values of several key hyperparameters in this method, e.g., μmax,ρ,γ,β\mu_{max}, \rho, \gamma, \beta, etc?
  2. Can the authors provide a detailed training setup?
  3. Do the gains hold with a different LoRA rank or model?
  4. How does this score compare with other metrics, e.g., cosine similarity?
5 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer 2Nuk (1/2)

We thank the reviewer for finding the problem valid and useful, for noting that the online control provides robustness along the training process, and for recognizing that FedSALT achieves the highest average accuracy with ablations which are sound.

Tables report client-average test accuracy (%) as mean ± standard deviation over three seeds.

W1 + W2

Order of updates. Algorithm 1 gives one communication round. In round tt, each client optimizes its local objective with the strengths μi,lt\mu_{i,l}^t and the anchor Mˇlt−1\check{M}_l^{t-1} received at the start of the round. After aggregation, the server scores each client update against the reference from round t−1t-1, updates the strengths to μi,lt+1\mu_{i,l}^{t+1}, and updates the momentum and reference to MltM_l^t, which are then used in round t+1t+1. We will state the motivation once, in Section 2.1, and present this order in one place in Section 3.5.

Subspace similarity in Figure 1. For client ii and an adapted module, let UiU_i and ViV_i be orthonormal bases of the column space and the row space of the client's LoRA update ΔWi=αrBiAi\Delta W_i=\frac{\alpha}{r}B_iA_i (Eq. (1)) after local training. The output-side and input-side similarities between clients ii and jj are κijL=∥Ui⊤Uj∥F2r,κijR=∥Vi⊤Vj∥F2r,\kappa^L_{ij}=\frac{\Vert U_i^\top U_j\Vert_F^2}{r},\qquad \kappa^R_{ij}=\frac{\Vert V_i^\top V_j\Vert_F^2}{r}, which lie in [0,1][0,1] and equal one exactly when the two subspaces coincide. Figure 1 reports their equal-weight average over the seven adapted projections (q, k, v, o, gate, up, down). We will add this definition to Section 2.1.

Notation. We will use "Eq." consistently throughout the paper.

W3 + Q4

Novelty. Prior work uses the asymmetry between the two LoRA factors to decide which factor to share or personalize; for example, FedSA-LoRA shares the input-side factor. FedSALT instead measures output-subspace disagreement on the realized update, which is invariant to the LoRA factorization, and converts it online into client- and module-specific control for the next round. FedSALT outperforms FedSA-LoRA on every benchmark (Table 1).

Simpler signals. Table 3 already compares the output-side score with the input-side score, the two-sided score, matched-mean uniform control, and permuted assignment. The table below adds the cosine similarity between each client update and the momentum reference as the control signal. The output-side subspace score is more accurate. Cosine similarity rewards alignment with a single direction, whereas the subspace score accepts any update within the shared consensus subspace and penalizes only the component that leaves it.

Control signalAccuracy (%)
FedSALT58.52 ± 0.14
Cosine similarity58.03 ± 0.49

W4

Matched regularization strength. The table below compares output-side and input-side sensing at matched average regularization strength, reported relative to FedSALT. At matched strength, output-side sensing remains more accurate than input-side sensing.

VariantAverage strength (relative to FedSALT)Accuracy (%)
FedSALT1.0058.52 ± 0.14
Input-side score0.9857.96 ± 0.29

Absolute off-subspace response. Proposition 1 directly characterizes the absolute off-subspace response: its squared norm decreases monotonically with the proximal strength through the response coefficient aK(μ)a_K(\mu).

Normalized off-subspace energy. FedSALT defines drift by direction, as the fraction of an update that lies outside the consensus subspace (Section 3.2). Absolute energies are not comparable across training runs, because each run measures them against its own consensus subspace and its updates differ in overall scale. The normalized energy removes this scale, which is why Table 3 compares it across variants.

How FedSALT reduces drift. The alignment score determines which client-module pairs are regulated and how strongly (Section 3.3), and the momentum-lookahead target determines the direction toward which they are pulled, namely the recent consensus trajectory (Section 3.4). Drifting modules are therefore steered toward the shared direction, while well-aligned modules remain lightly constrained.

AuthorsOpenReview ↗

Rebuttal to Reviewer 2Nuk (2/2)

W5 + Q3

LoRA rank. The main experiments use LoRA rank r=16r=16, as stated in Section 5.

Additional backbones. The table below reports MC-8 with Qwen2.5-1.5B and Llama-3.2-3B under the protocol of Section 5. FedSALT remains the most accurate method on both backbones.

MethodQwen2.5-1.5BLlama-3.2-3B
FedSALT68.76 ± 0.0867.48 ± 0.11
FedRot-LoRA68.44 ± 0.2167.38 ± 0.22
FedProx68.62 ± 0.3167.01 ± 0.13
SCAFFOLD68.44 ± 0.1567.10 ± 0.18
FedEx-LoRA68.53 ± 0.5067.38 ± 0.29
FedAvg68.57 ± 0.4267.34 ± 0.16
FedSA-LoRA67.94 ± 0.0667.18 ± 0.02
FFA-LoRA68.26 ± 0.2366.02 ± 0.04

More clients. MC-12 extends MC-8 to 12 clients. FedSALT remains the most accurate method with more clients.

MethodMC-12
FedSALT58.65 ± 0.08
FedRot-LoRA58.17 ± 0.17
FedProx58.60 ± 0.34
SCAFFOLD58.38 ± 0.33
FedEx-LoRA58.30 ± 0.19
FedAvg58.23 ± 0.12
FedSA-LoRA57.85 ± 0.16
FFA-LoRA57.07 ± 0.44

Size of the gain. The table below reports a paired comparison on MC-8 over three seeds. FedSALT is more accurate than every baseline on every seed, and the improvement is statistically significant against every baseline.

BaselineSeeds where FedSALT is higherOne-sided paired tt-test pp-value
FedRot-LoRA3/30.030
FedProx3/30.012
SCAFFOLD3/30.016
FedEx-LoRA3/31.1 × 10⁻⁴
FedAvg3/30.013
FedSA-LoRA3/30.0017
FFA-LoRA3/30.0026

Communication. FedSALT leaves the uplink unchanged and adds a single downlink broadcast of one low-rank reference shared by all clients, and its total communication remains below SCAFFOLD (Table 4). We further match wall-clock time on MC-12 by limiting FedSALT to the time FedAvg needs for its full training, computed from the per-round runtime in Table 4, while every baseline keeps its full training. Within this budget, FedSALT reaches the same accuracy as in the MC-12 table above and remains the most accurate method.

We will include these results in the revision.

Q1 + Q2

Hyperparameters. FedSALT uses μmax⁡=0.1\mu_{\max}=0.1, γ=0.9\gamma=0.9, ρ=0.9\rho=0.9, β=1.0\beta=1.0, rM=4r=64r_M=4r=64, and rref=16r_{\mathrm{ref}}=16.

Training setup. The training setup is given in Section 5 and Appendix D.2, and training runs for 60 communication rounds. We will add the hyperparameter values above to Appendix D.2.

AuthorsOpenReview ↗

Gentle Follow-up for Reviewer 2Nuk on Our Rebuttal

Dear Reviewer 2Nuk,

Thank you again for your review. As the discussion period closes today, we would like to briefly summarize our response.

  • Novelty and simpler signals. FedSALT measures output-subspace disagreement on the realized update, which is invariant to the LoRA factorization, and converts it online into client- and module-specific control. FedSALT outperforms FedSA-LoRA on every benchmark, and its output-side score is more accurate than cosine similarity and, at matched average regularization strength, than the input-side score.
  • Scope and size of the gain. FedSALT remains the most accurate method with Qwen2.5-1.5B and Llama-3.2-3B and with 12 clients, and it is more accurate than every baseline on every seed, with statistically significant improvements.
  • Communication. FedSALT leaves the uplink unchanged, its total communication remains below SCAFFOLD, and it remains the most accurate method under a matched wall-clock budget.
  • Presentation and setup. We clarified the order of updates and the definition of the subspace similarity in Figure 1, and listed all hyperparameter values.

We sincerely appreciate your time and would be grateful for any feedback on whether our response addresses your concerns. If it does, we would be grateful if you would kindly consider updating your rating.

Reviewer_7iNDOpenReview ↗
Rating
3
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper focuses on federated fine-tuning of large language models (LLMs) utilizing the heterogeneous federated LoRA technique. It first observes an empirical asymmetry in heterogeneous client tasks. Based on this, a novel geometry-aware closed-loop optimization framework is proposed, which could selectively guide drifting modules toward a momentum-lookahead consensus anchor while retaining standard factor-wise FedAvg aggregation. Experiments on three heterogeneous federated LLM benchmarks demonstrate the effectiveness of the proposed method.

Critical strengths

(+) This paper tackles federated fine-tuning of large language models (LLMs), which is a critical problem for deploying LLMs in real-world scenarios.

(+) The poposed FedSALT, a closed-loop optimization framework, is well-designed with strong theoretical support and good performance on three heterogeneous federated LLM benchmarks, compared with SOTA methods.

Critical weaknesses

(-) The formatting and presentation of this paper can be further improved, e.g., on page 1, line 46, the citations of Figure 1 and Figure 2 are too far away from the location of Figure 1 (page 3) and Figure 2 (page 4), and the symbols of “Eq.” and “equation” are used together, e.g., line 215, line 242, and line 323.

Questions for authors
  • Could you please provide more analysis or demonstration of the task differentiation between output-side geometry and input-side geometry to validate the observation?
3 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer 7iND

We thank the reviewer for recognizing that federated fine-tuning of LLMs is a critical problem for deploying LLMs in real-world scenarios, and that FedSALT is well-designed with strong theoretical support and achieves good performance on three heterogeneous federated LLM benchmarks.

The table below reports the mean output-side and input-side alignment scores defined in Section 3.2, where a higher score indicates closer alignment with the historical consensus subspace.

W1

Formatting. We will place Figures 1 and 2 next to their first citations and use "Eq." consistently throughout the paper.

Q1

Pairwise subspace similarity. Figure 1 compares every pair of clients directly. Output-side similarity varies substantially across client pairs, and the two MedMCQA clients, which share the same task, form the most similar pair on the output side. In contrast, input-side similarity is nearly identical across all pairs, so the input side carries little information about which task a client solves.

Alignment with the historical consensus. Figure 2 shows the same contrast across communication rounds: output-side alignment remains lower and more dispersed across clients than input-side alignment. Figure 3(a) further shows that output-side alignment is lower than input-side alignment for every individual client.

Additional federation. We further examine MC-12, which extends MC-8 to 12 clients. The same asymmetry holds, with output-side alignment remaining far below input-side alignment.

FederationOutput-side alignmentInput-side alignment
MC-120.3250.831

We will add this analysis to the appendix.

Thanks to authors’ rebuttal. I checked the rebuttal and other reviewers’ comments. I keep my original score.

AuthorsOpenReview ↗

Thank You for Maintaining Your Weak Accept Rating

We thank the reviewer for reading our response and the other reviews, and for maintaining the rating of Weak Accept.

We will incorporate the formatting improvements and the additional analysis of the output-side asymmetry into the revised paper.

Comments on the paper

1 comment
AuthorsOpenReview ↗

AC Rebuttal Summary: Additional Backbones, More Clients, and Paired Significance Tests

Dear Area Chair,

We thank you and the reviewers for the constructive discussion. We summarize the discussion with each reviewer below; each item refers to the corresponding W/Q in our response to that reviewer.

Reviewer 7iND (Rating 3: Weak acceptance)

Reviewer 7iND finds that the paper "tackles federated fine-tuning of large language models (LLMs), which is a critical problem for deploying LLMs in real-world scenarios" and that FedSALT "is well-designed with strong theoretical support and good performance on three heterogeneous federated LLM benchmarks, compared with SOTA methods." After checking our rebuttal and the other reviewers' comments, the reviewer kept the rating of "3: Weak acceptance."

Output-side asymmetry (Q1). The asymmetry appears in pairwise client similarity (Figure 1), in alignment across communication rounds (Figure 2), and for every individual client (Figure 3(a)), and it also holds on MC-12, which extends MC-8 to 12 clients (Section 5.2 and Appendix D.3).

Formatting (W1). Figures 1 and 2 now appear next to their first citations, and "Eq." is used consistently.

Reviewer 1Lia (Rating 2: Weak rejection)

Reviewer 1Lia highlights a "clear geometric motivation", noting that the paper "identifies stronger task differentiation in output-side than input-side update geometry and uses this observation to guide selective local regularization." The reviewer describes a "principled drift measurement": "the alignment score operates on realized adapter updates, making it invariant to equivalent LoRA factorizations. Computation remains in low-rank form without constructing dense updates." The reviewer also finds the "informative mechanism ablations" helpful to "distinguish the contributions of geometry-aware allocation and momentum guidance."

Theory (W1). Full participation and exact averaging are the standard setting of federated optimization analyses (Li et al., ICLR 2020; Stich, ICLR 2019). Our analysis is carried out in the realized-adapter space, where FedSALT senses and controls drift, and because FedSALT changes only the local objective while keeping factor-wise FedAvg, it isolates exactly FedSALT's contribution (Section 4).

Strength of the gains (W2). On MC-8, FedSALT is more accurate than every baseline on every seed, with statistically significant paired improvements (Appendix D.5). FedSALT also remains the most accurate method with Qwen2.5-1.5B, Llama-3.2-3B, and 12 clients (Table 1 and Table 1 (Backbones)).

Trade-offs (W3). Within the wall-clock time of FedAvg's full training, FedSALT remains the most accurate method on MC-12. It leaves the uplink unchanged, its total communication remains below SCAFFOLD (Section 5.2 and Appendix D.4), and it is robust to its controller parameters and reference rank (Appendix D.6).

Reviewer 2Nuk (Rating 2: Weak rejection)

Reviewer 2Nuk finds that "the problem the authors analyze is valid and useful", that "in federated LoRA, adjusting regularization can control local updates under different client data", and that "this online control provides robustness along the training process." The reviewer also notes that "the method achieves the highest average accuracy in the experiments, showing its benefits" and that the results "are over three random seeds and include uniform regularization with the same average strength and randomly assigned strengths, which are sound."

Novelty and simpler signals (W3 + Q4, W4). FedSALT measures output-subspace disagreement on the realized update, which is invariant to the LoRA factorization, and converts it online into client- and module-specific control. FedSALT outperforms FedSA-LoRA on every benchmark. Every simpler signal lowers accuracy: cosine similarity and the input-side score at matched average regularization strength (Appendix D.1), as well as the two-sided score, matched-mean uniform control, and permuted assignment (Table 3).

Scope and consistency of the gains (W5 + Q3). FedSALT is the most accurate method in every setting we evaluate: the three benchmarks, two additional backbones, and 12 clients, with statistically significant paired improvements on MC-8. It also remains the most accurate method under a matched wall-clock budget.

Presentation and setup (W1 + W2, Q1 + Q2). The revised paper states the motivation once, presents the order of updates in one place, defines the subspace similarity in Figure 1 (Sections 2.1 and 3.5), and lists all hyperparameter values (Appendix D.2).

We have revised the PDF accordingly, with all changes highlighted in blue. To keep the table numbers cited in the discussion unchanged, new tables placed among the existing ones are numbered as variants of an existing table, e.g., Table 1 (Backbones).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.